2024-03-19T02:25:48.497 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-19T02:25:48.497 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609439 --name pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi --owner scheduled_pdonnell@teuthology --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --verbose -- /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609439/orig.config.yaml 2024-03-19T02:25:48.550 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-19T02:25:48.643 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609439 branch: wip-batrick-testing-20240318.181317 description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: pdonnell@redhat.com first_in_suite: false job_id: '7609439' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240318.181317 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - PG_AVAILABILITY - PG_DEGRADED - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240318.181317 sha1: e48b1f6c0608b2615e1890cc46286c3877d6e910 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 3005 sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 sleep_before_teardown: 0 subset: 433/512 suite: fs suite_branch: wip-batrick-testing-20240318.181317 suite_path: /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: e48b1f6c0608b2615e1890cc46286c3877d6e910 targets: smithi059.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMzMI8XIydaJGNFQyK09I9223zuTu1APOjRa7nxNh8UVYxaUkDdjFfwzhCSeSg8i83zJoKQWjOtgE59qoumoPSU= smithi176.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOkpD1WsLZAYVh+6AZbSWS/oin4WSKn8n3/VavpxHJfc16KQb/GHcyvbt5NiraEW3cbNG1mGLUAbeI0HcK5TYeM= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: - /cephfs/home/pdonnell/src/github.com_batrick_ceph_wip-batrick-testing-20240318.181317/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml meta: {} postmerge: [] variables: fail_fs: false teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-18_21:05:49 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs false || true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-03-19T02:25:48.643 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa; will attempt to use it 2024-03-19T02:25:48.644 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks 2024-03-19T02:25:48.644 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-19T02:25:48.646 INFO:teuthology.task.internal:Checking packages... 2024-03-19T02:25:48.676 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b6ae24918b03b1b8a19c9263a50655bba80bb810' 2024-03-19T02:25:48.676 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-19T02:25:48.676 INFO:teuthology.packaging:ref: None 2024-03-19T02:25:48.676 INFO:teuthology.packaging:tag: None 2024-03-19T02:25:48.676 INFO:teuthology.packaging:branch: wip-batrick-testing-20240318.181317 2024-03-19T02:25:48.676 INFO:teuthology.packaging:sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:25:48.676 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240318.181317 2024-03-19T02:25:49.171 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2206.gb6ae2491 2024-03-19T02:25:49.173 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-19T02:25:49.179 INFO:teuthology.task.internal:no buildpackages task found 2024-03-19T02:25:49.180 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-19T02:25:49.218 INFO:teuthology.task.internal:Saving configuration 2024-03-19T02:25:49.245 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-19T02:25:49.253 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T02:25:49.280 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609439', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 02:20:59.477107', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMzMI8XIydaJGNFQyK09I9223zuTu1APOjRa7nxNh8UVYxaUkDdjFfwzhCSeSg8i83zJoKQWjOtgE59qoumoPSU='} 2024-03-19T02:25:49.299 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609439', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 02:20:59.478516', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOkpD1WsLZAYVh+6AZbSWS/oin4WSKn8n3/VavpxHJfc16KQb/GHcyvbt5NiraEW3cbNG1mGLUAbeI0HcK5TYeM='} 2024-03-19T02:25:49.300 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-19T02:25:49.310 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-03-19T02:25:49.311 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-03-19T02:25:49.311 INFO:teuthology.run_tasks:Running task console_log... 2024-03-19T02:25:49.427 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc0ee3ce160>, signals=[15]) 2024-03-19T02:25:49.427 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-19T02:25:49.436 INFO:teuthology.task.internal:Opening connections... 2024-03-19T02:25:49.436 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi059.front.sepia.ceph.com 2024-03-19T02:25:49.439 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:25:49.549 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi176.front.sepia.ceph.com 2024-03-19T02:25:49.550 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:25:49.662 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-19T02:25:49.672 DEBUG:teuthology.orchestra.run.smithi059:> uname -m 2024-03-19T02:25:49.690 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2024-03-19T02:25:49.691 DEBUG:teuthology.orchestra.run.smithi059:> cat /etc/os-release 2024-03-19T02:25:49.749 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="CentOS Stream" 2024-03-19T02:25:49.749 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="9" 2024-03-19T02:25:49.749 INFO:teuthology.orchestra.run.smithi059.stdout:ID="centos" 2024-03-19T02:25:49.749 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE="rhel fedora" 2024-03-19T02:25:49.750 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="9" 2024-03-19T02:25:49.750 INFO:teuthology.orchestra.run.smithi059.stdout:PLATFORM_ID="platform:el9" 2024-03-19T02:25:49.750 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-19T02:25:49.750 INFO:teuthology.orchestra.run.smithi059.stdout:ANSI_COLOR="0;31" 2024-03-19T02:25:49.750 INFO:teuthology.orchestra.run.smithi059.stdout:LOGO="fedora-logo-icon" 2024-03-19T02:25:49.750 INFO:teuthology.orchestra.run.smithi059.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-19T02:25:49.750 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://centos.org/" 2024-03-19T02:25:49.750 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-19T02:25:49.750 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-19T02:25:49.751 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-19T02:25:49.751 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2024-03-19T02:25:49.778 DEBUG:teuthology.orchestra.run.smithi176:> uname -m 2024-03-19T02:25:49.796 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2024-03-19T02:25:49.796 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2024-03-19T02:25:49.851 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="CentOS Stream" 2024-03-19T02:25:49.851 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="9" 2024-03-19T02:25:49.851 INFO:teuthology.orchestra.run.smithi176.stdout:ID="centos" 2024-03-19T02:25:49.851 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE="rhel fedora" 2024-03-19T02:25:49.851 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="9" 2024-03-19T02:25:49.852 INFO:teuthology.orchestra.run.smithi176.stdout:PLATFORM_ID="platform:el9" 2024-03-19T02:25:49.852 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-19T02:25:49.852 INFO:teuthology.orchestra.run.smithi176.stdout:ANSI_COLOR="0;31" 2024-03-19T02:25:49.852 INFO:teuthology.orchestra.run.smithi176.stdout:LOGO="fedora-logo-icon" 2024-03-19T02:25:49.852 INFO:teuthology.orchestra.run.smithi176.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-19T02:25:49.852 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://centos.org/" 2024-03-19T02:25:49.852 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-19T02:25:49.852 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-19T02:25:49.852 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-19T02:25:49.852 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2024-03-19T02:25:49.886 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-19T02:25:49.899 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-19T02:25:49.931 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-19T02:25:49.932 DEBUG:teuthology.orchestra.run.smithi059:> test '!' -e /home/ubuntu/cephtest 2024-03-19T02:25:49.935 DEBUG:teuthology.orchestra.run.smithi176:> test '!' -e /home/ubuntu/cephtest 2024-03-19T02:25:49.952 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-19T02:25:49.961 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-19T02:25:49.962 DEBUG:teuthology.orchestra.run.smithi059:> test -z $(ls -A /var/lib/ceph) 2024-03-19T02:25:49.992 DEBUG:teuthology.orchestra.run.smithi176:> test -z $(ls -A /var/lib/ceph) 2024-03-19T02:25:50.021 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-19T02:25:50.076 INFO:teuthology.run_tasks:Running task kernel... 2024-03-19T02:25:50.097 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-19T02:25:50.098 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-19T02:25:50.098 DEBUG:teuthology.orchestra.run.smithi059:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-19T02:25:50.099 DEBUG:teuthology.orchestra.run.smithi176:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-19T02:25:50.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:25:50.116 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-03-19T02:25:50.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:25:50.118 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-03-19T02:25:50.174 INFO:teuthology.orchestra.run.smithi176.stdout:5.14.0-361.el9.x86_64 2024-03-19T02:25:50.174 INFO:teuthology.task.kernel:Running kernel on smithi176: 5.14.0-361.el9.x86_64 2024-03-19T02:25:50.174 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum install -y kernel 2024-03-19T02:25:50.175 INFO:teuthology.orchestra.run.smithi059.stdout:5.14.0-361.el9.x86_64 2024-03-19T02:25:50.175 INFO:teuthology.task.kernel:Running kernel on smithi059: 5.14.0-361.el9.x86_64 2024-03-19T02:25:50.175 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum install -y kernel 2024-03-19T02:25:52.187 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-19T02:25:52.278 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-19T02:25:53.720 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-19T02:25:53.805 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-19T02:25:55.852 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 8.0 MB 00:01 2024-03-19T02:25:56.270 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.0 MB 00:02 2024-03-19T02:25:57.954 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-03-19T02:26:03.538 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 6.1 MB 00:02 2024-03-19T02:26:05.172 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-03-19T02:26:06.648 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-19T02:26:07.835 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - AppStream 1.7 MB/s | 19 MB 00:10 2024-03-19T02:26:11.623 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-19T02:26:13.721 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-19T02:26:13.721 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-19T02:26:13.804 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:26:13.811 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:26:13.812 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-03-19T02:26:13.812 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:26:13.812 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-19T02:26:13.812 INFO:teuthology.orchestra.run.smithi176.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-19T02:26:13.812 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-19T02:26:13.812 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-19T02:26:13.812 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-19T02:26:13.812 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-19T02:26:13.812 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:26:13.812 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-19T02:26:13.812 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:26:13.812 INFO:teuthology.orchestra.run.smithi176.stdout:Install 4 Packages 2024-03-19T02:26:13.813 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:26:13.813 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 76 M 2024-03-19T02:26:13.813 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 123 M 2024-03-19T02:26:13.813 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-19T02:26:14.224 INFO:teuthology.orchestra.run.smithi176.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 388 kB/s | 44 kB 00:00 2024-03-19T02:26:17.049 INFO:teuthology.orchestra.run.smithi176.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 5.1 MB/s | 15 MB 00:02 2024-03-19T02:26:18.132 INFO:teuthology.orchestra.run.smithi176.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 7.0 MB/s | 27 MB 00:03 2024-03-19T02:26:18.558 INFO:teuthology.orchestra.run.smithi176.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 7.5 MB/s | 33 MB 00:04 2024-03-19T02:26:18.558 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:26:18.558 INFO:teuthology.orchestra.run.smithi176.stdout:Total 16 MB/s | 76 MB 00:04 2024-03-19T02:26:19.223 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-19T02:26:19.602 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-19T02:26:19.602 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-19T02:26:20.600 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-19T02:26:20.600 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-19T02:26:23.069 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-19T02:26:23.686 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-19T02:26:23.773 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T02:26:26.514 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T02:26:26.649 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T02:26:31.368 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T02:26:31.424 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:26:36.131 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:27:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:27:05.197 INFO:teuthology.orchestra.run.smithi176.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-19T02:27:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:27:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:27:05.586 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:27:05.586 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-19T02:27:05.586 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T02:27:05.586 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T02:27:06.007 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:27:06.007 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:27:06.007 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-19T02:27:06.007 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-19T02:27:06.007 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-19T02:27:06.008 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-19T02:27:06.008 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-19T02:27:06.008 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:27:06.008 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:27:06.326 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2024-03-19T02:27:06.429 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-19T02:27:06.429 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-19T02:27:06.429 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-19T02:27:06.429 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-19T02:27:06.430 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-19T02:27:06.430 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-03-19T02:27:06.430 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi176.front.sepia.ceph.com, path=None, version=distro) 2024-03-19T02:27:06.430 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2024-03-19T02:27:06.509 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-19T02:27:06.510 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-19T02:27:06.510 DEBUG:teuthology.orchestra.run.smithi176:> sudo rpm -qi grub2-tools 2024-03-19T02:27:06.543 INFO:teuthology.orchestra.run.smithi176.stdout:Name : grub2-tools 2024-03-19T02:27:06.544 INFO:teuthology.orchestra.run.smithi176.stdout:Epoch : 1 2024-03-19T02:27:06.544 INFO:teuthology.orchestra.run.smithi176.stdout:Version : 2.06 2024-03-19T02:27:06.544 INFO:teuthology.orchestra.run.smithi176.stdout:Release : 68.el9 2024-03-19T02:27:06.544 INFO:teuthology.orchestra.run.smithi176.stdout:Architecture: x86_64 2024-03-19T02:27:06.544 INFO:teuthology.orchestra.run.smithi176.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-19T02:27:06.544 INFO:teuthology.orchestra.run.smithi176.stdout:Group : Unspecified 2024-03-19T02:27:06.544 INFO:teuthology.orchestra.run.smithi176.stdout:Size : 8264254 2024-03-19T02:27:06.544 INFO:teuthology.orchestra.run.smithi176.stdout:License : GPLv3+ 2024-03-19T02:27:06.545 INFO:teuthology.orchestra.run.smithi176.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-19T02:27:06.545 INFO:teuthology.orchestra.run.smithi176.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-19T02:27:06.545 INFO:teuthology.orchestra.run.smithi176.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-19T02:27:06.545 INFO:teuthology.orchestra.run.smithi176.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-19T02:27:06.545 INFO:teuthology.orchestra.run.smithi176.stdout:Packager : builder@centos.org 2024-03-19T02:27:06.545 INFO:teuthology.orchestra.run.smithi176.stdout:Vendor : CentOS 2024-03-19T02:27:06.545 INFO:teuthology.orchestra.run.smithi176.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-19T02:27:06.545 INFO:teuthology.orchestra.run.smithi176.stdout:Summary : Support tools for GRUB. 2024-03-19T02:27:06.545 INFO:teuthology.orchestra.run.smithi176.stdout:Description : 2024-03-19T02:27:06.545 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:27:06.545 INFO:teuthology.orchestra.run.smithi176.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-19T02:27:06.545 INFO:teuthology.orchestra.run.smithi176.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-19T02:27:06.545 INFO:teuthology.orchestra.run.smithi176.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-19T02:27:06.545 INFO:teuthology.orchestra.run.smithi176.stdout:hardware devices. 2024-03-19T02:27:06.546 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:27:06.546 INFO:teuthology.orchestra.run.smithi176.stdout:This subpackage provides tools for support of all platforms. 2024-03-19T02:27:06.546 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-19T02:27:06.547 INFO:teuthology.task.kernel:Updating grub on smithi176 to boot 5.14.0-430.el9.x86_64 2024-03-19T02:27:06.547 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-19T02:27:07.131 INFO:teuthology.orchestra.run.smithi176.stderr:Generating grub configuration file ... 2024-03-19T02:27:08.450 INFO:teuthology.orchestra.run.smithi176.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-19T02:27:08.487 INFO:teuthology.orchestra.run.smithi176.stderr:done 2024-03-19T02:27:08.490 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-03-19T02:27:08.505 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.gVxA6Fcy4v 2024-03-19T02:27:08.505 DEBUG:teuthology.orchestra.run.smithi176:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.gVxA6Fcy4v 2024-03-19T02:27:08.574 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.gVxA6Fcy4v 2024-03-19T02:27:08.710 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.gVxA6Fcy4v is 6KB 2024-03-19T02:27:08.729 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.gVxA6Fcy4v 2024-03-19T02:27:08.743 DEBUG:teuthology.orchestra.run.smithi176:> sudo /bin/ls /boot/loader/entries || true 2024-03-19T02:27:08.813 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-19T02:27:08.813 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-19T02:27:08.814 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-19T02:27:08.814 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-19T02:27:08.815 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-19T02:27:08.980 DEBUG:teuthology.orchestra.run.smithi176:> sudo shutdown -r now 2024-03-19T02:27:38.983 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-03-19T02:27:38.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T02:27:38.984 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:27:57.347 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T02:28:06.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T02:28:06.348 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:28:09.406 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T02:28:21.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T02:28:21.408 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:28:53.306 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-03-19T02:28:53.714 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T02:28:53.714 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-430.el9.x86_64"... 2024-03-19T02:28:53.715 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-03-19T02:28:53.729 INFO:teuthology.orchestra.run.smithi176.stdout:5.14.0-430.el9.x86_64 2024-03-19T02:28:53.729 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-19T02:28:53.729 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-19T02:28:53.729 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-19T02:28:54.730 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-03-19T02:28:54.731 DEBUG:teuthology.orchestra.run.smithi176:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-19T02:28:54.790 INFO:teuthology.orchestra.run.smithi176.stdout:ttyS1 2024-03-19T02:28:54.828 DEBUG:teuthology.parallel:result is None 2024-03-19T02:30:39.347 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - CRB 23 kB/s | 6.1 MB 04:27 2024-03-19T02:30:40.997 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-19T02:31:02.322 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 997 kB/s | 21 MB 00:21 2024-03-19T02:31:07.471 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-03-19T02:31:09.679 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-19T02:31:09.679 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-19T02:31:09.781 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T02:31:09.785 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T02:31:09.785 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-03-19T02:31:09.785 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T02:31:09.785 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-19T02:31:09.785 INFO:teuthology.orchestra.run.smithi059.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-19T02:31:09.785 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-03-19T02:31:09.785 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-19T02:31:09.786 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-19T02:31:09.786 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-19T02:31:09.786 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T02:31:09.786 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-19T02:31:09.786 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T02:31:09.786 INFO:teuthology.orchestra.run.smithi059.stdout:Install 4 Packages 2024-03-19T02:31:09.786 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T02:31:09.787 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 76 M 2024-03-19T02:31:09.787 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 123 M 2024-03-19T02:31:09.787 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-19T02:31:10.265 INFO:teuthology.orchestra.run.smithi059.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 125 kB/s | 44 kB 00:00 2024-03-19T02:31:22.006 INFO:teuthology.orchestra.run.smithi059.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 1.2 MB/s | 15 MB 00:12 2024-03-19T02:31:33.113 INFO:teuthology.orchestra.run.smithi059.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 1.2 MB/s | 27 MB 00:22 2024-03-19T02:31:38.438 INFO:teuthology.orchestra.run.smithi059.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 1.2 MB/s | 33 MB 00:28 2024-03-19T02:31:38.438 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:31:38.438 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.6 MB/s | 76 MB 00:28 2024-03-19T02:31:39.133 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-19T02:31:39.591 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-19T02:31:39.592 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-19T02:31:40.652 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-19T02:31:40.653 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-19T02:31:43.263 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-19T02:31:43.901 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-19T02:31:43.988 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T02:31:46.878 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T02:31:47.008 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T02:31:52.079 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T02:31:52.124 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:31:57.227 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:32:28.067 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:32:28.068 INFO:teuthology.orchestra.run.smithi059.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-19T02:32:28.068 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T02:32:28.113 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:32:28.417 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:32:28.417 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-19T02:32:28.417 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T02:32:28.417 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T02:32:28.794 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:32:28.794 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T02:32:28.794 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-19T02:32:28.794 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-19T02:32:28.795 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-19T02:32:28.795 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-19T02:32:28.795 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-19T02:32:28.795 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T02:32:28.795 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T02:32:29.132 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2024-03-19T02:32:29.280 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-19T02:32:29.281 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-19T02:32:29.281 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-19T02:32:29.281 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-19T02:32:29.281 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-19T02:32:29.281 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-03-19T02:32:29.281 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi059.front.sepia.ceph.com, path=None, version=distro) 2024-03-19T02:32:29.282 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2024-03-19T02:32:29.364 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-19T02:32:29.364 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-19T02:32:29.365 DEBUG:teuthology.orchestra.run.smithi059:> sudo rpm -qi grub2-tools 2024-03-19T02:32:29.402 INFO:teuthology.orchestra.run.smithi059.stdout:Name : grub2-tools 2024-03-19T02:32:29.402 INFO:teuthology.orchestra.run.smithi059.stdout:Epoch : 1 2024-03-19T02:32:29.402 INFO:teuthology.orchestra.run.smithi059.stdout:Version : 2.06 2024-03-19T02:32:29.402 INFO:teuthology.orchestra.run.smithi059.stdout:Release : 68.el9 2024-03-19T02:32:29.402 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: x86_64 2024-03-19T02:32:29.402 INFO:teuthology.orchestra.run.smithi059.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-19T02:32:29.403 INFO:teuthology.orchestra.run.smithi059.stdout:Group : Unspecified 2024-03-19T02:32:29.403 INFO:teuthology.orchestra.run.smithi059.stdout:Size : 8264254 2024-03-19T02:32:29.403 INFO:teuthology.orchestra.run.smithi059.stdout:License : GPLv3+ 2024-03-19T02:32:29.403 INFO:teuthology.orchestra.run.smithi059.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-19T02:32:29.403 INFO:teuthology.orchestra.run.smithi059.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-19T02:32:29.403 INFO:teuthology.orchestra.run.smithi059.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-19T02:32:29.403 INFO:teuthology.orchestra.run.smithi059.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-19T02:32:29.403 INFO:teuthology.orchestra.run.smithi059.stdout:Packager : builder@centos.org 2024-03-19T02:32:29.403 INFO:teuthology.orchestra.run.smithi059.stdout:Vendor : CentOS 2024-03-19T02:32:29.403 INFO:teuthology.orchestra.run.smithi059.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-19T02:32:29.403 INFO:teuthology.orchestra.run.smithi059.stdout:Summary : Support tools for GRUB. 2024-03-19T02:32:29.403 INFO:teuthology.orchestra.run.smithi059.stdout:Description : 2024-03-19T02:32:29.403 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T02:32:29.403 INFO:teuthology.orchestra.run.smithi059.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-19T02:32:29.403 INFO:teuthology.orchestra.run.smithi059.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-19T02:32:29.404 INFO:teuthology.orchestra.run.smithi059.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-19T02:32:29.404 INFO:teuthology.orchestra.run.smithi059.stdout:hardware devices. 2024-03-19T02:32:29.404 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T02:32:29.404 INFO:teuthology.orchestra.run.smithi059.stdout:This subpackage provides tools for support of all platforms. 2024-03-19T02:32:29.405 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-19T02:32:29.405 INFO:teuthology.task.kernel:Updating grub on smithi059 to boot 5.14.0-430.el9.x86_64 2024-03-19T02:32:29.405 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-19T02:32:30.056 INFO:teuthology.orchestra.run.smithi059.stderr:Generating grub configuration file ... 2024-03-19T02:32:31.473 INFO:teuthology.orchestra.run.smithi059.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-19T02:32:31.510 INFO:teuthology.orchestra.run.smithi059.stderr:done 2024-03-19T02:32:31.513 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2024-03-19T02:32:31.569 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.KHPCYyrFfi 2024-03-19T02:32:31.569 DEBUG:teuthology.orchestra.run.smithi059:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KHPCYyrFfi 2024-03-19T02:32:31.651 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.KHPCYyrFfi 2024-03-19T02:32:31.793 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.KHPCYyrFfi is 6KB 2024-03-19T02:32:31.842 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.KHPCYyrFfi 2024-03-19T02:32:31.857 DEBUG:teuthology.orchestra.run.smithi059:> sudo /bin/ls /boot/loader/entries || true 2024-03-19T02:32:31.925 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-19T02:32:31.925 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-19T02:32:31.925 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-19T02:32:31.925 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-19T02:32:31.927 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-19T02:32:32.072 DEBUG:teuthology.orchestra.run.smithi059:> sudo shutdown -r now 2024-03-19T02:33:02.074 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-03-19T02:33:02.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-19T02:33:02.076 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:33:20.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.59 2024-03-19T02:33:29.416 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-19T02:33:29.417 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:34:29.419 DEBUG:teuthology.orchestra.remote:timed out 2024-03-19T02:34:41.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-19T02:34:41.422 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:34:41.756 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-03-19T02:34:42.172 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-19T02:34:42.172 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-430.el9.x86_64"... 2024-03-19T02:34:42.172 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-03-19T02:34:42.187 INFO:teuthology.orchestra.run.smithi059.stdout:5.14.0-430.el9.x86_64 2024-03-19T02:34:42.187 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-19T02:34:42.187 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-19T02:34:42.187 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-19T02:34:43.188 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-03-19T02:34:43.188 DEBUG:teuthology.orchestra.run.smithi059:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-19T02:34:43.250 INFO:teuthology.orchestra.run.smithi059.stdout:ttyS1 2024-03-19T02:34:43.293 DEBUG:teuthology.parallel:result is None 2024-03-19T02:34:43.293 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-19T02:34:43.301 INFO:teuthology.task.internal:Creating test directory... 2024-03-19T02:34:43.301 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-19T02:34:43.303 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-19T02:34:43.322 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-19T02:34:43.337 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-19T02:34:43.370 INFO:teuthology.task.internal:Creating archive directory... 2024-03-19T02:34:43.370 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-19T02:34:43.373 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-19T02:34:43.410 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-19T02:34:43.417 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-19T02:34:43.417 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-19T02:34:43.433 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-19T02:34:43.470 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T02:34:43.484 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T02:34:43.492 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T02:34:43.506 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T02:34:43.508 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-19T02:34:43.515 INFO:teuthology.task.internal:Configuring sudo... 2024-03-19T02:34:43.515 DEBUG:teuthology.orchestra.run.smithi059:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-19T02:34:43.528 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-19T02:34:43.557 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-19T02:34:43.566 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-19T02:34:43.567 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-19T02:34:43.597 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-19T02:34:43.613 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T02:34:43.694 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T02:34:43.742 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T02:34:43.824 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T02:34:43.866 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-19T02:34:43.866 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-19T02:34:43.932 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T02:34:43.977 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T02:34:44.059 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T02:34:44.102 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T02:34:44.145 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-19T02:34:44.145 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-19T02:34:44.212 DEBUG:teuthology.orchestra.run.smithi059:> sudo service rsyslog restart 2024-03-19T02:34:44.215 DEBUG:teuthology.orchestra.run.smithi176:> sudo service rsyslog restart 2024-03-19T02:34:44.263 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T02:34:44.305 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T02:34:44.666 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-19T02:34:44.674 INFO:teuthology.task.internal:Starting timer... 2024-03-19T02:34:44.674 INFO:teuthology.run_tasks:Running task pcp... 2024-03-19T02:34:44.717 INFO:teuthology.run_tasks:Running task selinux... 2024-03-19T02:34:44.748 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-19T02:34:44.748 DEBUG:teuthology.orchestra.run.smithi059:> sudo service auditd rotate 2024-03-19T02:34:44.812 INFO:teuthology.orchestra.run.smithi059.stdout:Rotating logs: 2024-03-19T02:34:44.815 DEBUG:teuthology.orchestra.run.smithi176:> sudo service auditd rotate 2024-03-19T02:34:44.870 INFO:teuthology.orchestra.run.smithi176.stdout:Rotating logs: 2024-03-19T02:34:44.872 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-19T02:34:44.872 DEBUG:teuthology.orchestra.run.smithi059:> /usr/sbin/getenforce 2024-03-19T02:34:44.892 INFO:teuthology.orchestra.run.smithi059.stdout:Permissive 2024-03-19T02:34:44.892 DEBUG:teuthology.orchestra.run.smithi176:> /usr/sbin/getenforce 2024-03-19T02:34:44.926 INFO:teuthology.orchestra.run.smithi176.stdout:Permissive 2024-03-19T02:34:44.926 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi059.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi176.front.sepia.ceph.com': 'permissive'} 2024-03-19T02:34:44.926 DEBUG:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T02:34:44.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:34:44.960 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T02:34:44.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:34:44.993 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-19T02:34:44.993 DEBUG:teuthology.orchestra.run.smithi059:> sudo /usr/sbin/setenforce permissive 2024-03-19T02:34:45.037 DEBUG:teuthology.orchestra.run.smithi176:> sudo /usr/sbin/setenforce permissive 2024-03-19T02:34:45.070 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-19T02:34:45.080 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-19T02:34:45.102 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-19T02:34:45.105 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi059.front.sepia.ceph.com,smithi176.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-19T02:49:58.564 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi059.front.sepia.ceph.com'), Remote(name='ubuntu@smithi176.front.sepia.ceph.com')] 2024-03-19T02:49:58.586 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-19T02:49:58.587 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:49:58.677 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-03-19T02:49:58.753 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-19T02:49:58.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T02:49:58.754 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:49:58.844 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-03-19T02:49:58.920 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T02:49:58.921 INFO:teuthology.run_tasks:Running task clock... 2024-03-19T02:49:58.932 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-19T02:49:58.933 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-19T02:49:58.933 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T02:49:58.936 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-19T02:49:58.936 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T02:49:58.968 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-19T02:49:58.986 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-19T02:49:58.990 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-19T02:49:59.007 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-19T02:49:59.016 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: ntpd: command not found 2024-03-19T02:49:59.030 INFO:teuthology.orchestra.run.smithi059.stdout:506 Cannot talk to daemon 2024-03-19T02:49:59.035 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: ntpd: command not found 2024-03-19T02:49:59.047 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-19T02:49:59.049 INFO:teuthology.orchestra.run.smithi176.stdout:506 Cannot talk to daemon 2024-03-19T02:49:59.064 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-19T02:49:59.065 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-19T02:49:59.081 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-19T02:49:59.117 INFO:teuthology.orchestra.run.smithi059.stderr:bash: line 1: ntpq: command not found 2024-03-19T02:49:59.121 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T02:49:59.121 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2024-03-19T02:49:59.121 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:49:59.121 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:49:59.121 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:49:59.121 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:49:59.133 INFO:teuthology.orchestra.run.smithi176.stderr:bash: line 1: ntpq: command not found 2024-03-19T02:49:59.137 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T02:49:59.137 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-03-19T02:49:59.137 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:49:59.137 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:49:59.137 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:49:59.137 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:49:59.138 INFO:teuthology.run_tasks:Running task install... 2024-03-19T02:49:59.146 DEBUG:teuthology.task.install:project ceph 2024-03-19T02:49:59.147 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'}} 2024-03-19T02:49:59.147 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'} 2024-03-19T02:49:59.147 INFO:teuthology.task.install:Using flavor: default 2024-03-19T02:49:59.153 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-19T02:49:59.153 INFO:teuthology.task.install:extra packages: [] 2024-03-19T02:49:59.153 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'tag': None, 'wait_for_package': False} 2024-03-19T02:49:59.153 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-19T02:49:59.153 INFO:teuthology.packaging:ref: None 2024-03-19T02:49:59.154 INFO:teuthology.packaging:tag: None 2024-03-19T02:49:59.154 INFO:teuthology.packaging:branch: quincy 2024-03-19T02:49:59.154 INFO:teuthology.packaging:sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:49:59.154 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-03-19T02:49:59.156 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'tag': None, 'wait_for_package': False} 2024-03-19T02:49:59.156 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-19T02:49:59.156 INFO:teuthology.packaging:ref: None 2024-03-19T02:49:59.156 INFO:teuthology.packaging:tag: None 2024-03-19T02:49:59.156 INFO:teuthology.packaging:branch: quincy 2024-03-19T02:49:59.156 INFO:teuthology.packaging:sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:49:59.156 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-03-19T02:49:59.389 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/b4843bd759866f9ee1bdfb1dfa084ec7d085b910/centos/9/flavors/default/ 2024-03-19T02:49:59.390 INFO:teuthology.task.install.rpm:Package version is 17.2.7-700.gb4843bd7 2024-03-19T02:49:59.396 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/b4843bd759866f9ee1bdfb1dfa084ec7d085b910/centos/9/flavors/default/ 2024-03-19T02:49:59.396 INFO:teuthology.task.install.rpm:Package version is 17.2.7-700.gb4843bd7 2024-03-19T02:49:59.503 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/b4843bd759866f9ee1bdfb1dfa084ec7d085b910/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/b4843bd759866f9ee1bdfb1dfa084ec7d085b910/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/b4843bd759866f9ee1bdfb1dfa084ec7d085b910/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-19T02:49:59.503 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-19T02:49:59.503 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-19T02:49:59.510 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/b4843bd759866f9ee1bdfb1dfa084ec7d085b910/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/b4843bd759866f9ee1bdfb1dfa084ec7d085b910/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/b4843bd759866f9ee1bdfb1dfa084ec7d085b910/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-19T02:49:59.510 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-19T02:49:59.510 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-19T02:49:59.531 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-19T02:49:59.531 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-19T02:49:59.531 INFO:teuthology.packaging:ref: None 2024-03-19T02:49:59.531 INFO:teuthology.packaging:tag: None 2024-03-19T02:49:59.531 INFO:teuthology.packaging:branch: quincy 2024-03-19T02:49:59.531 INFO:teuthology.packaging:sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:49:59.532 DEBUG:teuthology.orchestra.run.smithi176:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-19T02:49:59.538 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-19T02:49:59.538 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-19T02:49:59.538 INFO:teuthology.packaging:ref: None 2024-03-19T02:49:59.538 INFO:teuthology.packaging:tag: None 2024-03-19T02:49:59.538 INFO:teuthology.packaging:branch: quincy 2024-03-19T02:49:59.538 INFO:teuthology.packaging:sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:49:59.539 DEBUG:teuthology.orchestra.run.smithi059:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-19T02:49:59.599 DEBUG:teuthology.orchestra.run.smithi176:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-19T02:49:59.606 DEBUG:teuthology.orchestra.run.smithi059:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-19T02:49:59.679 DEBUG:teuthology.orchestra.run.smithi176:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-19T02:49:59.686 DEBUG:teuthology.orchestra.run.smithi059:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-19T02:49:59.745 INFO:teuthology.orchestra.run.smithi176.stdout:check_obsoletes = 1 2024-03-19T02:49:59.747 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum clean all 2024-03-19T02:49:59.753 INFO:teuthology.orchestra.run.smithi059.stdout:check_obsoletes = 1 2024-03-19T02:49:59.754 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum clean all 2024-03-19T02:49:59.988 INFO:teuthology.orchestra.run.smithi176.stdout:74 files removed 2024-03-19T02:49:59.999 INFO:teuthology.orchestra.run.smithi059.stdout:74 files removed 2024-03-19T02:50:00.015 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-radosgw 2024-03-19T02:50:00.027 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-radosgw 2024-03-19T02:50:00.784 INFO:teuthology.orchestra.run.smithi059.stdout:ceph packages for x86_64 172 kB/s | 77 kB 00:00 2024-03-19T02:50:00.865 INFO:teuthology.orchestra.run.smithi176.stdout:ceph packages for x86_64 141 kB/s | 77 kB 00:00 2024-03-19T02:50:01.133 INFO:teuthology.orchestra.run.smithi059.stdout:ceph noarch packages 31 kB/s | 9.1 kB 00:00 2024-03-19T02:50:01.291 INFO:teuthology.orchestra.run.smithi176.stdout:ceph noarch packages 24 kB/s | 9.1 kB 00:00 2024-03-19T02:50:01.517 INFO:teuthology.orchestra.run.smithi059.stdout:ceph source packages 5.2 kB/s | 1.8 kB 00:00 2024-03-19T02:50:01.642 INFO:teuthology.orchestra.run.smithi176.stdout:ceph source packages 5.7 kB/s | 1.8 kB 00:00 2024-03-19T02:50:02.286 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - AppStream 9.2 MB/s | 6.7 MB 00:00 2024-03-19T02:50:02.352 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - AppStream 9.9 MB/s | 6.7 MB 00:00 2024-03-19T02:50:03.911 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - BaseOS 4.7 MB/s | 2.0 MB 00:00 2024-03-19T02:50:03.927 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - BaseOS 4.4 MB/s | 2.0 MB 00:00 2024-03-19T02:50:04.935 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-19T02:50:04.951 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-19T02:51:29.259 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - AppStream 232 kB/s | 19 MB 01:23 2024-03-19T02:51:34.217 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.1 MB 00:01 2024-03-19T02:51:36.093 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-03-19T02:51:38.021 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-19T02:51:42.927 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-19T02:51:45.346 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:51:45.394 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:51:45.394 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-19T02:51:45.395 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:51:45.395 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-19T02:51:45.395 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 14 M 2024-03-19T02:51:45.395 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrading: 2024-03-19T02:51:45.395 INFO:teuthology.orchestra.run.smithi176.stdout: librados2 x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 3.1 M 2024-03-19T02:51:45.395 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1 x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 2.9 M 2024-03-19T02:51:45.395 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-19T02:51:45.395 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 5.0 M 2024-03-19T02:51:45.395 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 23 M 2024-03-19T02:51:45.395 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 25 k 2024-03-19T02:51:45.395 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-19T02:51:45.396 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2 x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 636 k 2024-03-19T02:51:45.396 INFO:teuthology.orchestra.run.smithi176.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-19T02:51:45.396 INFO:teuthology.orchestra.run.smithi176.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-19T02:51:45.396 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-19T02:51:45.396 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1 x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 458 k 2024-03-19T02:51:45.396 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-19T02:51:45.396 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2 x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 4.9 M 2024-03-19T02:51:45.396 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-19T02:51:45.396 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-19T02:51:45.396 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 45 k 2024-03-19T02:51:45.396 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 118 k 2024-03-19T02:51:45.396 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 156 k 2024-03-19T02:51:45.396 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-19T02:51:45.397 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 320 k 2024-03-19T02:51:45.397 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 294 k 2024-03-19T02:51:45.397 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 99 k 2024-03-19T02:51:45.397 INFO:teuthology.orchestra.run.smithi176.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-19T02:51:45.397 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:51:45.397 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-19T02:51:45.397 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:51:45.397 INFO:teuthology.orchestra.run.smithi176.stdout:Install 22 Packages 2024-03-19T02:51:45.397 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrade 2 Packages 2024-03-19T02:51:45.397 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:51:45.398 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 58 M 2024-03-19T02:51:45.398 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-19T02:51:46.062 INFO:teuthology.orchestra.run.smithi176.stdout:(1/24): ceph-base-17.2.7-700.gb4843bd7.el9.x86_ 9.7 MB/s | 5.0 MB 00:00 2024-03-19T02:51:46.103 INFO:teuthology.orchestra.run.smithi176.stdout:(2/24): ceph-selinux-17.2.7-700.gb4843bd7.el9.x 606 kB/s | 25 kB 00:00 2024-03-19T02:51:46.228 INFO:teuthology.orchestra.run.smithi176.stdout:(3/24): libcephfs2-17.2.7-700.gb4843bd7.el9.x86 5.0 MB/s | 636 kB 00:00 2024-03-19T02:51:46.287 INFO:teuthology.orchestra.run.smithi176.stdout:(4/24): libradosstriper1-17.2.7-700.gb4843bd7.e 7.7 MB/s | 458 kB 00:00 2024-03-19T02:51:46.646 INFO:teuthology.orchestra.run.smithi176.stdout:(5/24): librgw2-17.2.7-700.gb4843bd7.el9.x86_64 14 MB/s | 4.9 MB 00:00 2024-03-19T02:51:46.687 INFO:teuthology.orchestra.run.smithi176.stdout:(6/24): python3-ceph-argparse-17.2.7-700.gb4843 1.1 MB/s | 45 kB 00:00 2024-03-19T02:51:46.738 INFO:teuthology.orchestra.run.smithi176.stdout:(7/24): python3-ceph-common-17.2.7-700.gb4843bd 2.3 MB/s | 118 kB 00:00 2024-03-19T02:51:46.796 INFO:teuthology.orchestra.run.smithi176.stdout:(8/24): python3-cephfs-17.2.7-700.gb4843bd7.el9 2.6 MB/s | 156 kB 00:00 2024-03-19T02:51:46.863 INFO:teuthology.orchestra.run.smithi176.stdout:(9/24): python3-rados-17.2.7-700.gb4843bd7.el9. 4.7 MB/s | 320 kB 00:00 2024-03-19T02:51:46.922 INFO:teuthology.orchestra.run.smithi176.stdout:(10/24): python3-rbd-17.2.7-700.gb4843bd7.el9.x 4.9 MB/s | 294 kB 00:00 2024-03-19T02:51:46.972 INFO:teuthology.orchestra.run.smithi176.stdout:(11/24): python3-rgw-17.2.7-700.gb4843bd7.el9.x 1.9 MB/s | 99 kB 00:00 2024-03-19T02:51:47.141 INFO:teuthology.orchestra.run.smithi176.stdout:(12/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 954 kB/s | 159 kB 00:00 2024-03-19T02:51:47.198 INFO:teuthology.orchestra.run.smithi176.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 759 kB/s | 44 kB 00:00 2024-03-19T02:51:47.290 INFO:teuthology.orchestra.run.smithi176.stdout:(14/24): librdkafka-1.6.1-102.el9.x86_64.rpm 7.1 MB/s | 661 kB 00:00 2024-03-19T02:51:47.331 INFO:teuthology.orchestra.run.smithi176.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-19T02:51:47.374 INFO:teuthology.orchestra.run.smithi176.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 985 kB/s | 41 kB 00:00 2024-03-19T02:51:47.774 INFO:teuthology.orchestra.run.smithi176.stdout:(17/24): ceph-common-17.2.7-700.gb4843bd7.el9.x 10 MB/s | 23 MB 00:02 2024-03-19T02:51:47.974 INFO:teuthology.orchestra.run.smithi176.stdout:(18/24): ceph-radosgw-17.2.7-700.gb4843bd7.el9. 5.8 MB/s | 14 MB 00:02 2024-03-19T02:51:48.358 INFO:teuthology.orchestra.run.smithi176.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 83 kB/s | 49 kB 00:00 2024-03-19T02:51:48.408 INFO:teuthology.orchestra.run.smithi176.stdout:(20/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 299 kB/s | 309 kB 00:01 2024-03-19T02:51:48.441 INFO:teuthology.orchestra.run.smithi176.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 152 kB/s | 67 kB 00:00 2024-03-19T02:51:48.742 INFO:teuthology.orchestra.run.smithi176.stdout:(22/24): librbd1-17.2.7-700.gb4843bd7.el9.x86_6 9.8 MB/s | 2.9 MB 00:00 2024-03-19T02:51:48.800 INFO:teuthology.orchestra.run.smithi176.stdout:(23/24): librados2-17.2.7-700.gb4843bd7.el9.x86 7.8 MB/s | 3.1 MB 00:00 2024-03-19T02:51:48.867 INFO:teuthology.orchestra.run.smithi176.stdout:(24/24): thrift-0.15.0-2.el9.x86_64.rpm 3.1 MB/s | 1.6 MB 00:00 2024-03-19T02:51:48.873 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:51:48.873 INFO:teuthology.orchestra.run.smithi176.stdout:Total 17 MB/s | 58 MB 00:03 2024-03-19T02:51:48.873 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-19T02:51:48.906 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-19T02:51:48.906 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-19T02:51:49.357 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-19T02:51:49.358 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-19T02:51:50.618 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-19T02:51:50.941 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-03-19T02:51:51.003 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librados2-2:17.2.7-700.gb4843bd7.el9.x86_64 2/26 2024-03-19T02:51:51.102 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-2:17.2.7-700.gb4843bd7.el9.x86_64 2/26 2024-03-19T02:51:51.187 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rados-2:17.2.7-700.gb4843bd7.el9.x86_64 3/26 2024-03-19T02:51:51.213 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs2-2:17.2.7-700.gb4843bd7.el9.x86_64 4/26 2024-03-19T02:51:51.287 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephfs2-2:17.2.7-700.gb4843bd7.el9.x86_64 4/26 2024-03-19T02:51:51.329 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-03-19T02:51:51.359 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-03-19T02:51:51.786 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-03-19T02:51:51.815 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librgw2-2:17.2.7-700.gb4843bd7.el9.x86_64 8/26 2024-03-19T02:51:51.877 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librgw2-2:17.2.7-700.gb4843bd7.el9.x86_64 8/26 2024-03-19T02:51:51.907 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-03-19T02:51:51.949 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-argparse-2:17.2.7-700.gb4843bd7.el9.x 10/26 2024-03-19T02:51:51.986 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cephfs-2:17.2.7-700.gb4843bd7.el9.x86_64 11/26 2024-03-19T02:51:52.048 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rgw-2:17.2.7-700.gb4843bd7.el9.x86_64 12/26 2024-03-19T02:51:52.076 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libradosstriper1-2:17.2.7-700.gb4843bd7.el9.x86_64 13/26 2024-03-19T02:51:52.125 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libradosstriper1-2:17.2.7-700.gb4843bd7.el9.x86_64 13/26 2024-03-19T02:51:52.182 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-03-19T02:51:52.226 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-03-19T02:51:52.258 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-03-19T02:51:52.553 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-03-19T02:51:52.588 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librbd1-2:17.2.7-700.gb4843bd7.el9.x86_64 18/26 2024-03-19T02:51:52.677 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-2:17.2.7-700.gb4843bd7.el9.x86_64 18/26 2024-03-19T02:51:52.752 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rbd-2:17.2.7-700.gb4843bd7.el9.x86_64 19/26 2024-03-19T02:51:52.782 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-common-2:17.2.7-700.gb4843bd7.el9.x86 20/26 2024-03-19T02:51:55.289 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:17.2.7-700.gb4843bd7.el9.x86_64 21/26 2024-03-19T02:51:55.329 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-common-2:17.2.7-700.gb4843bd7.el9.x86_64 21/26 2024-03-19T02:51:55.853 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:17.2.7-700.gb4843bd7.el9.x86_64 21/26 2024-03-19T02:51:55.898 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-base-2:17.2.7-700.gb4843bd7.el9.x86_64 22/26 2024-03-19T02:51:55.988 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-base-2:17.2.7-700.gb4843bd7.el9.x86_64 22/26 2024-03-19T02:51:55.988 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-19T02:51:55.989 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-19T02:51:55.989 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:51:56.033 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-selinux-2:17.2.7-700.gb4843bd7.el9.x86_64 23/26 2024-03-19T02:52:05.595 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-selinux-2:17.2.7-700.gb4843bd7.el9.x86_64 23/26 2024-03-19T02:52:05.595 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /sys 2024-03-19T02:52:05.595 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /proc 2024-03-19T02:52:05.595 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /mnt 2024-03-19T02:52:05.595 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /var/tmp 2024-03-19T02:52:05.595 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /home 2024-03-19T02:52:05.595 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /root 2024-03-19T02:52:05.595 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /tmp 2024-03-19T02:52:05.595 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:06.868 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-radosgw-2:17.2.7-700.gb4843bd7.el9.x86_64 24/26 2024-03-19T02:52:06.929 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-700.gb4843bd7.el9.x86_64 24/26 2024-03-19T02:52:06.929 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:52:06.929 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-19T02:52:06.929 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T02:52:06.929 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T02:52:06.929 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:06.931 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-03-19T02:52:07.012 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-03-19T02:52:07.013 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-03-19T02:52:08.216 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-03-19T02:52:08.216 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-base-2:17.2.7-700.gb4843bd7.el9.x86_64 1/26 2024-03-19T02:52:08.216 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-common-2:17.2.7-700.gb4843bd7.el9.x86_64 2/26 2024-03-19T02:52:08.216 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-radosgw-2:17.2.7-700.gb4843bd7.el9.x86_64 3/26 2024-03-19T02:52:08.216 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-selinux-2:17.2.7-700.gb4843bd7.el9.x86_64 4/26 2024-03-19T02:52:08.216 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs2-2:17.2.7-700.gb4843bd7.el9.x86_64 5/26 2024-03-19T02:52:08.216 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libradosstriper1-2:17.2.7-700.gb4843bd7.el9.x86_64 6/26 2024-03-19T02:52:08.216 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librgw2-2:17.2.7-700.gb4843bd7.el9.x86_64 7/26 2024-03-19T02:52:08.216 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-argparse-2:17.2.7-700.gb4843bd7.el9.x 8/26 2024-03-19T02:52:08.216 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-common-2:17.2.7-700.gb4843bd7.el9.x86 9/26 2024-03-19T02:52:08.217 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cephfs-2:17.2.7-700.gb4843bd7.el9.x86_64 10/26 2024-03-19T02:52:08.217 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rados-2:17.2.7-700.gb4843bd7.el9.x86_64 11/26 2024-03-19T02:52:08.217 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rbd-2:17.2.7-700.gb4843bd7.el9.x86_64 12/26 2024-03-19T02:52:08.217 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rgw-2:17.2.7-700.gb4843bd7.el9.x86_64 13/26 2024-03-19T02:52:08.217 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-03-19T02:52:08.217 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-03-19T02:52:08.217 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-03-19T02:52:08.217 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-03-19T02:52:08.217 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-03-19T02:52:08.217 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-03-19T02:52:08.217 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-03-19T02:52:08.217 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-03-19T02:52:08.217 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-03-19T02:52:08.217 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-2:17.2.7-700.gb4843bd7.el9.x86_64 23/26 2024-03-19T02:52:08.217 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-03-19T02:52:08.218 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-2:17.2.7-700.gb4843bd7.el9.x86_64 25/26 2024-03-19T02:52:09.232 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-03-19T02:52:09.232 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:09.233 INFO:teuthology.orchestra.run.smithi176.stdout:Upgraded: 2024-03-19T02:52:09.233 INFO:teuthology.orchestra.run.smithi176.stdout: librados2-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:09.233 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:09.233 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-19T02:52:09.233 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:09.233 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:09.233 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:09.233 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:09.233 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-19T02:52:09.233 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:09.233 INFO:teuthology.orchestra.run.smithi176.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-19T02:52:09.233 INFO:teuthology.orchestra.run.smithi176.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-19T02:52:09.233 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-19T02:52:09.234 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:09.234 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-19T02:52:09.234 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:09.234 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-19T02:52:09.234 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-19T02:52:09.234 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:09.234 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:09.234 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:09.234 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-19T02:52:09.234 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:09.234 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:09.234 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:09.234 INFO:teuthology.orchestra.run.smithi176.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-19T02:52:09.234 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:09.235 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:52:09.563 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-test 2024-03-19T02:52:10.124 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:28 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:52:10.279 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:52:10.279 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:52:10.279 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-19T02:52:10.280 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:52:10.280 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-19T02:52:10.280 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 48 M 2024-03-19T02:52:10.280 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-19T02:52:10.280 INFO:teuthology.orchestra.run.smithi176.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-19T02:52:10.280 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 163 k 2024-03-19T02:52:10.280 INFO:teuthology.orchestra.run.smithi176.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-19T02:52:10.280 INFO:teuthology.orchestra.run.smithi176.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-19T02:52:10.280 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-19T02:52:10.280 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:10.280 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-19T02:52:10.280 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:52:10.280 INFO:teuthology.orchestra.run.smithi176.stdout:Install 6 Packages 2024-03-19T02:52:10.281 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:10.281 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 49 M 2024-03-19T02:52:10.281 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 196 M 2024-03-19T02:52:10.281 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-19T02:52:10.550 INFO:teuthology.orchestra.run.smithi176.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-19T02:52:10.600 INFO:teuthology.orchestra.run.smithi176.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-19T02:52:10.625 INFO:teuthology.orchestra.run.smithi176.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-19T02:52:10.675 INFO:teuthology.orchestra.run.smithi176.stdout:(4/6): libcephsqlite-17.2.7-700.gb4843bd7.el9.x 589 kB/s | 163 kB 00:00 2024-03-19T02:52:10.759 INFO:teuthology.orchestra.run.smithi176.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-19T02:52:12.743 INFO:teuthology.orchestra.run.smithi176.stdout:(6/6): ceph-test-17.2.7-700.gb4843bd7.el9.x86_6 20 MB/s | 48 MB 00:02 2024-03-19T02:52:12.745 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:52:12.746 INFO:teuthology.orchestra.run.smithi176.stdout:Total 20 MB/s | 49 MB 00:02 2024-03-19T02:52:12.760 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-19T02:52:12.780 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-19T02:52:12.780 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-19T02:52:13.188 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-19T02:52:13.188 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-19T02:52:13.763 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-19T02:52:13.839 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-19T02:52:13.879 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-19T02:52:13.908 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-19T02:52:13.951 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-19T02:52:13.979 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephsqlite-2:17.2.7-700.gb4843bd7.el9.x86_64 5/6 2024-03-19T02:52:18.441 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephsqlite-2:17.2.7-700.gb4843bd7.el9.x86_64 5/6 2024-03-19T02:52:18.512 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-test-2:17.2.7-700.gb4843bd7.el9.x86_64 6/6 2024-03-19T02:52:20.696 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-test-2:17.2.7-700.gb4843bd7.el9.x86_64 6/6 2024-03-19T02:52:20.696 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-test-2:17.2.7-700.gb4843bd7.el9.x86_64 1/6 2024-03-19T02:52:20.696 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephsqlite-2:17.2.7-700.gb4843bd7.el9.x86_64 2/6 2024-03-19T02:52:20.696 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-19T02:52:20.696 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-19T02:52:20.696 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-19T02:52:21.093 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-19T02:52:21.093 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:21.093 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-19T02:52:21.093 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:21.093 INFO:teuthology.orchestra.run.smithi176.stdout: jq-1.6-16.el9.x86_64 2024-03-19T02:52:21.093 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:21.093 INFO:teuthology.orchestra.run.smithi176.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-19T02:52:21.093 INFO:teuthology.orchestra.run.smithi176.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-19T02:52:21.093 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-19T02:52:21.093 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:21.093 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:52:21.369 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph 2024-03-19T02:52:21.939 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:39 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:52:22.097 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:52:22.100 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-19T02:52:22.100 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-19T02:52:22.100 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-19T02:52:22.100 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-19T02:52:22.100 INFO:teuthology.orchestra.run.smithi176.stdout: ceph x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 6.5 k 2024-03-19T02:52:22.100 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-19T02:52:22.100 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 2.1 M 2024-03-19T02:52:22.100 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 1.4 M 2024-03-19T02:52:22.100 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core noarch 2:17.2.7-700.gb4843bd7.el9 ceph-noarch 239 k 2024-03-19T02:52:22.100 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 3.7 M 2024-03-19T02:52:22.100 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 15 M 2024-03-19T02:52:22.100 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-19T02:52:22.101 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-19T02:52:22.101 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-19T02:52:22.101 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-19T02:52:22.101 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-19T02:52:22.101 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-19T02:52:22.101 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-19T02:52:22.101 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-19T02:52:22.101 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-19T02:52:22.101 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-19T02:52:22.101 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-19T02:52:22.101 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-19T02:52:22.101 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-19T02:52:22.101 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-19T02:52:22.101 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-19T02:52:22.101 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-19T02:52:22.102 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-19T02:52:22.102 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-19T02:52:22.102 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-19T02:52:22.102 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-19T02:52:22.102 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-19T02:52:22.102 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-19T02:52:22.102 INFO:teuthology.orchestra.run.smithi176.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-19T02:52:22.102 INFO:teuthology.orchestra.run.smithi176.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-19T02:52:22.102 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-19T02:52:22.102 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-19T02:52:22.102 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-19T02:52:22.102 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:22.102 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-19T02:52:22.102 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-19T02:52:22.102 INFO:teuthology.orchestra.run.smithi176.stdout:Install 33 Packages 2024-03-19T02:52:22.103 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:22.103 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 27 M 2024-03-19T02:52:22.103 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 100 M 2024-03-19T02:52:22.103 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-19T02:52:22.410 INFO:teuthology.orchestra.run.smithi176.stdout:(1/33): ceph-17.2.7-700.gb4843bd7.el9.x86_64.rp 38 kB/s | 6.5 kB 00:00 2024-03-19T02:52:22.736 INFO:teuthology.orchestra.run.smithi176.stdout:(2/33): ceph-mon-17.2.7-700.gb4843bd7.el9.x86_6 11 MB/s | 3.7 MB 00:00 2024-03-19T02:52:22.961 INFO:teuthology.orchestra.run.smithi176.stdout:(3/33): ceph-mds-17.2.7-700.gb4843bd7.el9.x86_6 2.9 MB/s | 2.1 MB 00:00 2024-03-19T02:52:23.070 INFO:teuthology.orchestra.run.smithi176.stdout:(4/33): ceph-mgr-modules-core-17.2.7-700.gb4843 2.2 MB/s | 239 kB 00:00 2024-03-19T02:52:23.178 INFO:teuthology.orchestra.run.smithi176.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 144 kB 00:00 2024-03-19T02:52:23.212 INFO:teuthology.orchestra.run.smithi176.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 951 kB/s | 32 kB 00:00 2024-03-19T02:52:23.246 INFO:teuthology.orchestra.run.smithi176.stdout:(7/33): ceph-mgr-17.2.7-700.gb4843bd7.el9.x86_6 1.4 MB/s | 1.4 MB 00:01 2024-03-19T02:52:23.271 INFO:teuthology.orchestra.run.smithi176.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 800 kB/s | 47 kB 00:00 2024-03-19T02:52:23.296 INFO:teuthology.orchestra.run.smithi176.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 767 kB/s | 39 kB 00:00 2024-03-19T02:52:23.322 INFO:teuthology.orchestra.run.smithi176.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-19T02:52:23.372 INFO:teuthology.orchestra.run.smithi176.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-19T02:52:23.397 INFO:teuthology.orchestra.run.smithi176.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-19T02:52:23.422 INFO:teuthology.orchestra.run.smithi176.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-19T02:52:23.448 INFO:teuthology.orchestra.run.smithi176.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-19T02:52:23.598 INFO:teuthology.orchestra.run.smithi176.stdout:(15/33): ceph-osd-17.2.7-700.gb4843bd7.el9.x86_ 18 MB/s | 15 MB 00:00 2024-03-19T02:52:23.832 INFO:teuthology.orchestra.run.smithi176.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 113 kB/s | 43 kB 00:00 2024-03-19T02:52:23.948 INFO:teuthology.orchestra.run.smithi176.stdout:(17/33): python3-requests-2.25.1-8.el9.noarch.r 238 kB/s | 125 kB 00:00 2024-03-19T02:52:23.965 INFO:teuthology.orchestra.run.smithi176.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 501 kB/s | 172 kB 00:00 2024-03-19T02:52:24.066 INFO:teuthology.orchestra.run.smithi176.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-03-19T02:52:24.091 INFO:teuthology.orchestra.run.smithi176.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 141 kB/s | 18 kB 00:00 2024-03-19T02:52:24.124 INFO:teuthology.orchestra.run.smithi176.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 402 kB/s | 23 kB 00:00 2024-03-19T02:52:24.141 INFO:teuthology.orchestra.run.smithi176.stdout:(22/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 55 kB/s | 11 kB 00:00 2024-03-19T02:52:24.158 INFO:teuthology.orchestra.run.smithi176.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 289 kB/s | 19 kB 00:00 2024-03-19T02:52:24.183 INFO:teuthology.orchestra.run.smithi176.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 334 kB/s | 20 kB 00:00 2024-03-19T02:52:24.242 INFO:teuthology.orchestra.run.smithi176.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 943 kB/s | 79 kB 00:00 2024-03-19T02:52:24.267 INFO:teuthology.orchestra.run.smithi176.stdout:(26/33): python3-logutils-0.3.5-21.el9.noarch.r 367 kB/s | 46 kB 00:00 2024-03-19T02:52:24.293 INFO:teuthology.orchestra.run.smithi176.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-19T02:52:24.318 INFO:teuthology.orchestra.run.smithi176.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 217 kB/s | 16 kB 00:00 2024-03-19T02:52:24.368 INFO:teuthology.orchestra.run.smithi176.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 475 kB/s | 36 kB 00:00 2024-03-19T02:52:24.393 INFO:teuthology.orchestra.run.smithi176.stdout:(30/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 716 kB/s | 90 kB 00:00 2024-03-19T02:52:24.427 INFO:teuthology.orchestra.run.smithi176.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-19T02:52:24.469 INFO:teuthology.orchestra.run.smithi176.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 265 kB/s | 20 kB 00:00 2024-03-19T02:52:24.494 INFO:teuthology.orchestra.run.smithi176.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-03-19T02:52:24.500 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:52:24.500 INFO:teuthology.orchestra.run.smithi176.stdout:Total 11 MB/s | 27 MB 00:02 2024-03-19T02:52:24.513 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-19T02:52:24.551 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-19T02:52:24.551 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-19T02:52:24.885 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-19T02:52:24.885 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-19T02:52:26.118 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-19T02:52:26.180 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-19T02:52:26.214 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-19T02:52:26.248 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-19T02:52:26.340 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-19T02:52:26.417 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-19T02:52:26.458 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-19T02:52:26.530 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-19T02:52:26.584 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-19T02:52:26.622 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-19T02:52:26.671 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-19T02:52:26.730 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-19T02:52:26.918 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-19T02:52:26.960 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-19T02:52:26.996 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-19T02:52:27.033 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-19T02:52:27.063 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-19T02:52:27.140 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-19T02:52:27.173 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-19T02:52:27.198 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-19T02:52:27.239 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-19T02:52:27.273 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-19T02:52:27.300 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-19T02:52:27.388 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-19T02:52:27.422 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-19T02:52:27.502 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-19T02:52:27.647 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-19T02:52:27.764 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-19T02:52:27.914 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-700.gb4843bd7.el9.n 28/33 2024-03-19T02:52:27.944 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-2:17.2.7-700.gb4843bd7.el9.x86_64 29/33 2024-03-19T02:52:27.969 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-2:17.2.7-700.gb4843bd7.el9.x86_64 29/33 2024-03-19T02:52:27.969 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:52:27.969 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-19T02:52:27.969 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T02:52:27.969 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T02:52:27.969 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:29.354 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-osd-2:17.2.7-700.gb4843bd7.el9.x86_64 30/33 2024-03-19T02:52:29.381 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-osd-2:17.2.7-700.gb4843bd7.el9.x86_64 30/33 2024-03-19T02:52:29.382 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:52:29.382 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-19T02:52:29.382 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T02:52:29.382 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T02:52:29.382 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:29.771 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mon-2:17.2.7-700.gb4843bd7.el9.x86_64 31/33 2024-03-19T02:52:29.795 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mon-2:17.2.7-700.gb4843bd7.el9.x86_64 31/33 2024-03-19T02:52:29.795 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:52:29.795 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-19T02:52:29.795 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T02:52:29.795 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T02:52:29.795 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:29.988 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mds-2:17.2.7-700.gb4843bd7.el9.x86_64 32/33 2024-03-19T02:52:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mds-2:17.2.7-700.gb4843bd7.el9.x86_64 32/33 2024-03-19T02:52:30.012 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:52:30.012 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-19T02:52:30.012 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T02:52:30.012 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T02:52:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:30.070 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-2:17.2.7-700.gb4843bd7.el9.x86_64 33/33 2024-03-19T02:52:31.789 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-2:17.2.7-700.gb4843bd7.el9.x86_64 33/33 2024-03-19T02:52:31.789 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-2:17.2.7-700.gb4843bd7.el9.x86_64 1/33 2024-03-19T02:52:31.789 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mds-2:17.2.7-700.gb4843bd7.el9.x86_64 2/33 2024-03-19T02:52:31.789 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-2:17.2.7-700.gb4843bd7.el9.x86_64 3/33 2024-03-19T02:52:31.789 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mon-2:17.2.7-700.gb4843bd7.el9.x86_64 4/33 2024-03-19T02:52:31.789 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-osd-2:17.2.7-700.gb4843bd7.el9.x86_64 5/33 2024-03-19T02:52:31.790 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-700.gb4843bd7.el9.n 6/33 2024-03-19T02:52:31.790 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-19T02:52:31.790 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-19T02:52:31.790 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-19T02:52:31.790 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-19T02:52:31.790 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-19T02:52:31.790 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-19T02:52:31.790 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-19T02:52:31.790 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-19T02:52:31.790 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-19T02:52:31.790 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-19T02:52:31.790 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-19T02:52:31.790 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-19T02:52:31.790 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-19T02:52:31.791 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-19T02:52:31.791 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-19T02:52:31.791 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-19T02:52:31.791 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-19T02:52:31.791 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-19T02:52:31.791 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-19T02:52:31.791 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-19T02:52:31.791 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-19T02:52:31.791 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-19T02:52:31.791 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-19T02:52:31.791 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-19T02:52:31.792 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-19T02:52:31.792 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-19T02:52:32.796 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-19T02:52:32.796 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:32.797 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-19T02:52:32.797 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:32.797 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:32.797 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:32.797 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core-2:17.2.7-700.gb4843bd7.el9.noarch 2024-03-19T02:52:32.797 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:32.797 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:32.797 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-19T02:52:32.797 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-19T02:52:32.797 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-19T02:52:32.797 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-19T02:52:32.797 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-19T02:52:32.797 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-19T02:52:32.797 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-19T02:52:32.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-19T02:52:32.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-19T02:52:32.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-19T02:52:32.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-19T02:52:32.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-19T02:52:32.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-19T02:52:32.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-19T02:52:32.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-19T02:52:32.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ply-3.11-14.el9.noarch 2024-03-19T02:52:32.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-19T02:52:32.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-19T02:52:32.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-19T02:52:32.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-19T02:52:32.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-19T02:52:32.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-19T02:52:32.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-19T02:52:32.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-19T02:52:32.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-19T02:52:32.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-19T02:52:32.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-19T02:52:32.799 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:32.799 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:52:33.142 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-base 2024-03-19T02:52:33.725 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:51 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:52:33.822 INFO:teuthology.orchestra.run.smithi176.stdout:Package ceph-base-2:17.2.7-700.gb4843bd7.el9.x86_64 is already installed. 2024-03-19T02:52:33.875 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:52:33.876 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-19T02:52:33.876 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:52:33.938 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install cephadm 2024-03-19T02:52:34.509 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:52 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:52:34.657 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:52:34.658 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:52:34.658 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-19T02:52:34.658 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:52:34.658 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-19T02:52:34.658 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm noarch 2:17.2.7-700.gb4843bd7.el9 ceph-noarch 82 k 2024-03-19T02:52:34.658 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:34.658 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-19T02:52:34.658 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:52:34.659 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-19T02:52:34.659 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:34.659 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 82 k 2024-03-19T02:52:34.659 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 372 k 2024-03-19T02:52:34.659 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-19T02:52:34.871 INFO:teuthology.orchestra.run.smithi176.stdout:cephadm-17.2.7-700.gb4843bd7.el9.noarch.rpm 387 kB/s | 82 kB 00:00 2024-03-19T02:52:34.871 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:52:34.872 INFO:teuthology.orchestra.run.smithi176.stdout:Total 384 kB/s | 82 kB 00:00 2024-03-19T02:52:34.872 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-19T02:52:34.877 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-19T02:52:34.877 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-19T02:52:34.879 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-19T02:52:34.879 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-19T02:52:34.948 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-19T02:52:35.384 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephadm-2:17.2.7-700.gb4843bd7.el9.noarch 1/1 2024-03-19T02:52:35.475 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cephadm-2:17.2.7-700.gb4843bd7.el9.noarch 1/1 2024-03-19T02:52:35.815 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephadm-2:17.2.7-700.gb4843bd7.el9.noarch 1/1 2024-03-19T02:52:36.112 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cephadm-2:17.2.7-700.gb4843bd7.el9.noarch 1/1 2024-03-19T02:52:36.113 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:36.113 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-19T02:52:36.113 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm-2:17.2.7-700.gb4843bd7.el9.noarch 2024-03-19T02:52:36.113 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:36.113 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:52:36.301 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-immutable-object-cache 2024-03-19T02:52:36.874 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:54 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:52:37.029 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:52:37.029 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:52:37.029 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-19T02:52:37.030 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:52:37.030 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-19T02:52:37.030 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 137 k 2024-03-19T02:52:37.030 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:37.030 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-19T02:52:37.030 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:52:37.030 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-19T02:52:37.030 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:37.030 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 137 k 2024-03-19T02:52:37.030 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 379 k 2024-03-19T02:52:37.030 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-19T02:52:37.244 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-immutable-object-cache-17.2.7-700.gb4843bd 645 kB/s | 137 kB 00:00 2024-03-19T02:52:37.244 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:52:37.244 INFO:teuthology.orchestra.run.smithi176.stdout:Total 640 kB/s | 137 kB 00:00 2024-03-19T02:52:37.245 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-19T02:52:37.254 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-19T02:52:37.254 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-19T02:52:37.303 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-19T02:52:37.303 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-19T02:52:37.440 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-19T02:52:37.511 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-700.gb4843bd7.e 1/1 2024-03-19T02:52:37.535 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-700.gb4843bd7.e 1/1 2024-03-19T02:52:37.535 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:52:37.536 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-19T02:52:37.536 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:38.536 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-700.gb4843bd7.e 1/1 2024-03-19T02:52:38.537 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:38.537 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-19T02:52:38.537 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:52:38.537 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:38.537 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:52:38.790 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr 2024-03-19T02:52:39.370 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:57 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:52:39.467 INFO:teuthology.orchestra.run.smithi176.stdout:Package ceph-mgr-2:17.2.7-700.gb4843bd7.el9.x86_64 is already installed. 2024-03-19T02:52:39.519 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:52:39.520 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-19T02:52:39.520 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:52:39.591 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-dashboard 2024-03-19T02:52:40.152 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:58 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:52:40.301 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:52:40.302 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:52:40.302 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-19T02:52:40.302 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:52:40.302 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-19T02:52:40.302 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard noarch 2:17.2.7-700.gb4843bd7.el9 ceph-noarch 1.7 M 2024-03-19T02:52:40.302 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-19T02:52:40.302 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards noarch 2:17.2.7-700.gb4843bd7.el9 ceph-noarch 23 k 2024-03-19T02:52:40.302 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts noarch 2:17.2.7-700.gb4843bd7.el9 ceph-noarch 15 k 2024-03-19T02:52:40.302 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-03-19T02:52:40.303 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-19T02:52:40.303 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-19T02:52:40.303 INFO:teuthology.orchestra.run.smithi176.stdout:Installing weak dependencies: 2024-03-19T02:52:40.303 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-03-19T02:52:40.303 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:40.303 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-19T02:52:40.303 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:52:40.303 INFO:teuthology.orchestra.run.smithi176.stdout:Install 7 Packages 2024-03-19T02:52:40.303 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:40.304 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 2.0 M 2024-03-19T02:52:40.304 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 8.1 M 2024-03-19T02:52:40.304 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-19T02:52:40.648 INFO:teuthology.orchestra.run.smithi176.stdout:(1/7): ceph-grafana-dashboards-17.2.7-700.gb484 122 kB/s | 23 kB 00:00 2024-03-19T02:52:40.748 INFO:teuthology.orchestra.run.smithi176.stdout:(2/7): ceph-prometheus-alerts-17.2.7-700.gb4843 50 kB/s | 15 kB 00:00 2024-03-19T02:52:40.864 INFO:teuthology.orchestra.run.smithi176.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 42 kB/s | 9.0 kB 00:00 2024-03-19T02:52:40.940 INFO:teuthology.orchestra.run.smithi176.stdout:(4/7): ceph-mgr-dashboard-17.2.7-700.gb4843bd7. 3.5 MB/s | 1.7 MB 00:00 2024-03-19T02:52:40.973 INFO:teuthology.orchestra.run.smithi176.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 283 kB/s | 31 kB 00:00 2024-03-19T02:52:41.032 INFO:teuthology.orchestra.run.smithi176.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 144 kB/s | 41 kB 00:00 2024-03-19T02:52:41.324 INFO:teuthology.orchestra.run.smithi176.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 491 kB/s | 188 kB 00:00 2024-03-19T02:52:41.328 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:52:41.328 INFO:teuthology.orchestra.run.smithi176.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-03-19T02:52:41.328 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-19T02:52:41.336 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-19T02:52:41.336 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-19T02:52:41.397 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-19T02:52:41.398 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-19T02:52:41.692 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-19T02:52:41.750 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-03-19T02:52:41.776 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-03-19T02:52:41.822 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-03-19T02:52:41.854 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-03-19T02:52:41.883 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-700.gb4843bd7.el9.no 5/7 2024-03-19T02:52:42.110 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-700.gb4843bd7.el9.n 6/7 2024-03-19T02:52:42.153 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-700.gb4843bd7.el9.noarch 7/7 2024-03-19T02:52:42.687 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-700.gb4843bd7.el9.noarch 7/7 2024-03-19T02:52:42.687 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-700.gb4843bd7.el9.n 1/7 2024-03-19T02:52:42.687 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-700.gb4843bd7.el9.noarch 2/7 2024-03-19T02:52:42.687 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-700.gb4843bd7.el9.no 3/7 2024-03-19T02:52:42.687 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-03-19T02:52:42.687 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-03-19T02:52:42.687 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-03-19T02:52:42.987 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-03-19T02:52:42.988 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:42.988 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-19T02:52:42.988 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards-2:17.2.7-700.gb4843bd7.el9.noarch 2024-03-19T02:52:42.988 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard-2:17.2.7-700.gb4843bd7.el9.noarch 2024-03-19T02:52:42.988 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts-2:17.2.7-700.gb4843bd7.el9.noarch 2024-03-19T02:52:42.988 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-03-19T02:52:42.988 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-03-19T02:52:42.988 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-19T02:52:42.988 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-19T02:52:42.988 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:42.988 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:52:43.221 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-19T02:52:43.781 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:01 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:52:43.933 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:52:43.934 INFO:teuthology.orchestra.run.smithi176.stdout:=========================================================================================== 2024-03-19T02:52:43.934 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-19T02:52:43.934 INFO:teuthology.orchestra.run.smithi176.stdout:=========================================================================================== 2024-03-19T02:52:43.934 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-19T02:52:43.935 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-700.gb4843bd7.el9 ceph-noarch 7.4 M 2024-03-19T02:52:43.935 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-19T02:52:43.935 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-19T02:52:43.935 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-19T02:52:43.935 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-19T02:52:43.935 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-19T02:52:43.935 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-19T02:52:43.935 INFO:teuthology.orchestra.run.smithi176.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-19T02:52:43.935 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-19T02:52:43.935 INFO:teuthology.orchestra.run.smithi176.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-19T02:52:43.935 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-19T02:52:43.935 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-19T02:52:43.935 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-19T02:52:43.935 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:43.936 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-19T02:52:43.936 INFO:teuthology.orchestra.run.smithi176.stdout:=========================================================================================== 2024-03-19T02:52:43.936 INFO:teuthology.orchestra.run.smithi176.stdout:Install 12 Packages 2024-03-19T02:52:43.936 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:52:43.936 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 37 M 2024-03-19T02:52:43.937 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 209 M 2024-03-19T02:52:43.937 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-19T02:52:44.273 INFO:teuthology.orchestra.run.smithi176.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 216 kB/s | 32 kB 00:00 2024-03-19T02:52:44.306 INFO:teuthology.orchestra.run.smithi176.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 536 kB/s | 17 kB 00:00 2024-03-19T02:52:44.348 INFO:teuthology.orchestra.run.smithi176.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 835 kB/s | 34 kB 00:00 2024-03-19T02:52:44.415 INFO:teuthology.orchestra.run.smithi176.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-03-19T02:52:44.599 INFO:teuthology.orchestra.run.smithi176.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 19 MB/s | 4.7 MB 00:00 2024-03-19T02:52:44.665 INFO:teuthology.orchestra.run.smithi176.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-19T02:52:44.691 INFO:teuthology.orchestra.run.smithi176.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-19T02:52:44.866 INFO:teuthology.orchestra.run.smithi176.stdout:(8/12): ceph-mgr-diskprediction-local-17.2.7-70 10 MB/s | 7.4 MB 00:00 2024-03-19T02:52:45.133 INFO:teuthology.orchestra.run.smithi176.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-03-19T02:52:45.308 INFO:teuthology.orchestra.run.smithi176.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-03-19T02:54:42.214 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - AppStream 70 kB/s | 19 MB 04:36 2024-03-19T02:54:46.433 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-19T02:54:48.323 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 16 kB 00:00 2024-03-19T02:54:49.458 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-19T02:54:53.902 INFO:teuthology.orchestra.run.smithi176.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T02:54:53.902 INFO:teuthology.orchestra.run.smithi176.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T02:54:54.472 INFO:teuthology.orchestra.run.smithi176.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.5 kB/s | 195 kB 02:09 2024-03-19T02:54:54.525 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 46 kB/s | 1.7 kB 00:00 2024-03-19T02:54:54.922 INFO:teuthology.orchestra.run.smithi176.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 6.2 kB/s | 803 kB 02:10 2024-03-19T02:54:54.923 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:54:54.923 INFO:teuthology.orchestra.run.smithi176.stdout:Total 290 kB/s | 37 MB 02:10 2024-03-19T02:54:54.963 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-19T02:54:54.989 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-19T02:54:54.989 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-19T02:54:55.470 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-19T02:54:55.471 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-19T02:54:56.180 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-19T02:54:56.251 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-19T02:54:56.285 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-19T02:54:56.378 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-19T02:54:56.421 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-19T02:54:56.608 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-19T02:54:56.637 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-19T02:54:56.715 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-19T02:54:57.004 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T02:54:57.017 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T02:54:57.017 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-03-19T02:54:57.018 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T02:54:57.018 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-19T02:54:57.018 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 14 M 2024-03-19T02:54:57.018 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrading: 2024-03-19T02:54:57.018 INFO:teuthology.orchestra.run.smithi059.stdout: librados2 x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 3.1 M 2024-03-19T02:54:57.018 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1 x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 2.9 M 2024-03-19T02:54:57.018 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-03-19T02:54:57.018 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 5.0 M 2024-03-19T02:54:57.018 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 23 M 2024-03-19T02:54:57.018 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 25 k 2024-03-19T02:54:57.018 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-19T02:54:57.018 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2 x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 636 k 2024-03-19T02:54:57.019 INFO:teuthology.orchestra.run.smithi059.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-19T02:54:57.019 INFO:teuthology.orchestra.run.smithi059.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-19T02:54:57.019 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-19T02:54:57.019 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1 x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 458 k 2024-03-19T02:54:57.019 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-19T02:54:57.019 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2 x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 4.9 M 2024-03-19T02:54:57.019 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-19T02:54:57.019 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-19T02:54:57.019 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 45 k 2024-03-19T02:54:57.019 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 118 k 2024-03-19T02:54:57.019 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 156 k 2024-03-19T02:54:57.019 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-19T02:54:57.020 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 320 k 2024-03-19T02:54:57.020 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 294 k 2024-03-19T02:54:57.020 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 99 k 2024-03-19T02:54:57.020 INFO:teuthology.orchestra.run.smithi059.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-19T02:54:57.020 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T02:54:57.020 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-19T02:54:57.020 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T02:54:57.020 INFO:teuthology.orchestra.run.smithi059.stdout:Install 22 Packages 2024-03-19T02:54:57.020 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrade 2 Packages 2024-03-19T02:54:57.020 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T02:54:57.021 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 58 M 2024-03-19T02:54:57.021 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-19T02:54:57.365 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-19T02:54:57.473 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-19T02:54:57.684 INFO:teuthology.orchestra.run.smithi059.stdout:(1/24): ceph-base-17.2.7-700.gb4843bd7.el9.x86_ 9.8 MB/s | 5.0 MB 00:00 2024-03-19T02:54:57.718 INFO:teuthology.orchestra.run.smithi059.stdout:(2/24): ceph-selinux-17.2.7-700.gb4843bd7.el9.x 750 kB/s | 25 kB 00:00 2024-03-19T02:54:57.785 INFO:teuthology.orchestra.run.smithi059.stdout:(3/24): libcephfs2-17.2.7-700.gb4843bd7.el9.x86 9.3 MB/s | 636 kB 00:00 2024-03-19T02:54:57.843 INFO:teuthology.orchestra.run.smithi059.stdout:(4/24): libradosstriper1-17.2.7-700.gb4843bd7.e 7.7 MB/s | 458 kB 00:00 2024-03-19T02:54:58.191 INFO:teuthology.orchestra.run.smithi059.stdout:(5/24): librgw2-17.2.7-700.gb4843bd7.el9.x86_64 14 MB/s | 4.9 MB 00:00 2024-03-19T02:54:58.244 INFO:teuthology.orchestra.run.smithi059.stdout:(6/24): python3-ceph-argparse-17.2.7-700.gb4843 865 kB/s | 45 kB 00:00 2024-03-19T02:54:58.294 INFO:teuthology.orchestra.run.smithi059.stdout:(7/24): python3-ceph-common-17.2.7-700.gb4843bd 2.3 MB/s | 118 kB 00:00 2024-03-19T02:54:58.361 INFO:teuthology.orchestra.run.smithi059.stdout:(8/24): python3-cephfs-17.2.7-700.gb4843bd7.el9 2.3 MB/s | 156 kB 00:00 2024-03-19T02:54:58.428 INFO:teuthology.orchestra.run.smithi059.stdout:(9/24): python3-rados-17.2.7-700.gb4843bd7.el9. 4.7 MB/s | 320 kB 00:00 2024-03-19T02:54:58.495 INFO:teuthology.orchestra.run.smithi059.stdout:(10/24): python3-rbd-17.2.7-700.gb4843bd7.el9.x 4.3 MB/s | 294 kB 00:00 2024-03-19T02:54:58.554 INFO:teuthology.orchestra.run.smithi059.stdout:(11/24): python3-rgw-17.2.7-700.gb4843bd7.el9.x 1.7 MB/s | 99 kB 00:00 2024-03-19T02:54:58.670 INFO:teuthology.orchestra.run.smithi059.stdout:(12/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-19T02:54:58.703 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-19T02:54:58.704 INFO:teuthology.orchestra.run.smithi059.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-19T02:54:58.762 INFO:teuthology.orchestra.run.smithi059.stdout:(14/24): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-19T02:54:58.804 INFO:teuthology.orchestra.run.smithi059.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-19T02:54:58.838 INFO:teuthology.orchestra.run.smithi059.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-19T02:54:59.505 INFO:teuthology.orchestra.run.smithi059.stdout:(17/24): ceph-common-17.2.7-700.gb4843bd7.el9.x 9.7 MB/s | 23 MB 00:02 2024-03-19T02:54:59.730 INFO:teuthology.orchestra.run.smithi059.stdout:(18/24): ceph-radosgw-17.2.7-700.gb4843bd7.el9. 5.5 MB/s | 14 MB 00:02 2024-03-19T02:54:59.737 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-19T02:54:59.778 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-700.gb4843b 12/12 2024-03-19T02:55:01.105 INFO:teuthology.orchestra.run.smithi059.stdout:(19/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 136 kB/s | 309 kB 00:02 2024-03-19T02:55:01.172 INFO:teuthology.orchestra.run.smithi059.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 29 kB/s | 49 kB 00:01 2024-03-19T02:55:01.248 INFO:teuthology.orchestra.run.smithi059.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 44 kB/s | 67 kB 00:01 2024-03-19T02:55:01.456 INFO:teuthology.orchestra.run.smithi059.stdout:(22/24): librados2-17.2.7-700.gb4843bd7.el9.x86 11 MB/s | 3.1 MB 00:00 2024-03-19T02:55:01.682 INFO:teuthology.orchestra.run.smithi059.stdout:(23/24): librbd1-17.2.7-700.gb4843bd7.el9.x86_6 6.8 MB/s | 2.9 MB 00:00 2024-03-19T02:55:02.268 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-700.gb4843b 12/12 2024-03-19T02:55:02.268 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-700.gb4843b 1/12 2024-03-19T02:55:02.268 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-19T02:55:02.268 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-19T02:55:02.268 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-19T02:55:02.268 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-19T02:55:02.268 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-19T02:55:02.268 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-19T02:55:02.268 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-19T02:55:02.268 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-19T02:55:02.269 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-19T02:55:02.269 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-19T02:55:02.709 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-19T02:55:02.709 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:02.709 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-19T02:55:02.709 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local-2:17.2.7-700.gb4843bd7.el9.noarch 2024-03-19T02:55:02.709 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-19T02:55:02.709 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-19T02:55:02.709 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-19T02:55:02.709 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-19T02:55:02.709 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-19T02:55:02.709 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-19T02:55:02.710 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-19T02:55:02.710 INFO:teuthology.orchestra.run.smithi176.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-19T02:55:02.710 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-19T02:55:02.710 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-19T02:55:02.710 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-19T02:55:02.710 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:02.710 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:55:03.081 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-rook 2024-03-19T02:55:03.656 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:21 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:55:03.806 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:55:03.807 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================= 2024-03-19T02:55:03.807 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-19T02:55:03.807 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================= 2024-03-19T02:55:03.807 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-19T02:55:03.807 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook noarch 2:17.2.7-700.gb4843bd7.el9 ceph-noarch 47 k 2024-03-19T02:55:03.807 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-19T02:55:03.807 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-19T02:55:03.807 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-19T02:55:03.808 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-19T02:55:03.808 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-19T02:55:03.808 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-19T02:55:03.808 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-19T02:55:03.808 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-19T02:55:03.808 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-19T02:55:03.808 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-19T02:55:03.808 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-19T02:55:03.808 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-19T02:55:03.808 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-19T02:55:03.808 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:03.808 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-19T02:55:03.808 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================= 2024-03-19T02:55:03.808 INFO:teuthology.orchestra.run.smithi176.stdout:Install 13 Packages 2024-03-19T02:55:03.809 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:03.809 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 2.0 M 2024-03-19T02:55:03.809 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 26 M 2024-03-19T02:55:03.809 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-19T02:55:03.987 INFO:teuthology.orchestra.run.smithi176.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 195 kB/s | 24 kB 00:00 2024-03-19T02:55:04.012 INFO:teuthology.orchestra.run.smithi176.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 120 kB/s | 18 kB 00:00 2024-03-19T02:55:04.062 INFO:teuthology.orchestra.run.smithi176.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-19T02:55:04.087 INFO:teuthology.orchestra.run.smithi176.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-19T02:55:04.112 INFO:teuthology.orchestra.run.smithi176.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-19T02:55:04.138 INFO:teuthology.orchestra.run.smithi176.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-19T02:55:04.163 INFO:teuthology.orchestra.run.smithi176.stdout:(7/13): ceph-mgr-rook-17.2.7-700.gb4843bd7.el9. 157 kB/s | 47 kB 00:00 2024-03-19T02:55:04.572 INFO:teuthology.orchestra.run.smithi176.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 70 kB/s | 32 kB 00:00 2024-03-19T02:55:04.597 INFO:teuthology.orchestra.run.smithi176.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 33 kB/s | 15 kB 00:00 2024-03-19T02:55:04.681 INFO:teuthology.orchestra.run.smithi176.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 709 kB/s | 59 kB 00:00 2024-03-19T02:55:04.706 INFO:teuthology.orchestra.run.smithi176.stdout:(11/13): python3-google-auth-2.28.0-1.el9.noarc 373 kB/s | 203 kB 00:00 2024-03-19T02:55:04.731 INFO:teuthology.orchestra.run.smithi176.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-03-19T02:55:04.864 INFO:teuthology.orchestra.run.smithi176.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.5 MB/s | 1.0 MB 00:00 2024-03-19T02:55:04.869 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:55:04.869 INFO:teuthology.orchestra.run.smithi176.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-03-19T02:55:04.869 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-19T02:55:04.881 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-19T02:55:04.882 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-19T02:55:04.969 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-19T02:55:04.970 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-19T02:55:05.482 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-19T02:55:05.643 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-19T02:55:05.691 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-19T02:55:05.745 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-19T02:55:05.775 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-19T02:55:05.824 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-19T02:55:05.917 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-19T02:55:06.025 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-19T02:55:06.076 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-19T02:55:06.788 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-19T02:55:06.797 INFO:teuthology.orchestra.run.smithi059.stdout:(24/24): thrift-0.15.0-2.el9.x86_64.rpm 285 kB/s | 1.6 MB 00:05 2024-03-19T02:55:06.802 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:55:06.802 INFO:teuthology.orchestra.run.smithi059.stdout:Total 5.9 MB/s | 58 MB 00:09 2024-03-19T02:55:06.802 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-19T02:55:06.831 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-19T02:55:06.840 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-19T02:55:06.840 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-19T02:55:06.857 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-19T02:55:06.902 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-19T02:55:06.927 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-rook-2:17.2.7-700.gb4843bd7.el9.noarch 13/13 2024-03-19T02:55:07.312 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-19T02:55:07.313 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-19T02:55:07.697 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-700.gb4843bd7.el9.noarch 13/13 2024-03-19T02:55:07.697 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-rook-2:17.2.7-700.gb4843bd7.el9.noarch 1/13 2024-03-19T02:55:07.697 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-19T02:55:07.697 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-19T02:55:07.697 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-19T02:55:07.697 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-19T02:55:07.698 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-19T02:55:07.698 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-19T02:55:07.698 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-19T02:55:07.698 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-19T02:55:07.698 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-19T02:55:07.698 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-19T02:55:07.698 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-19T02:55:08.065 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-19T02:55:08.065 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:08.065 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-19T02:55:08.065 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook-2:17.2.7-700.gb4843bd7.el9.noarch 2024-03-19T02:55:08.065 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-19T02:55:08.066 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-19T02:55:08.066 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-19T02:55:08.066 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-19T02:55:08.066 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-19T02:55:08.066 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-19T02:55:08.066 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-19T02:55:08.066 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-19T02:55:08.066 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-19T02:55:08.066 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-19T02:55:08.066 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-19T02:55:08.066 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-19T02:55:08.066 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:08.066 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:55:08.305 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-cephadm 2024-03-19T02:55:08.570 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-19T02:55:08.863 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:26 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:55:08.915 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-03-19T02:55:08.967 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librados2-2:17.2.7-700.gb4843bd7.el9.x86_64 2/26 2024-03-19T02:55:09.014 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:55:09.015 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================= 2024-03-19T02:55:09.015 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-19T02:55:09.015 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================= 2024-03-19T02:55:09.015 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-19T02:55:09.015 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm noarch 2:17.2.7-700.gb4843bd7.el9 ceph-noarch 123 k 2024-03-19T02:55:09.015 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-19T02:55:09.015 INFO:teuthology.orchestra.run.smithi176.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-19T02:55:09.015 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-19T02:55:09.015 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-19T02:55:09.016 INFO:teuthology.orchestra.run.smithi176.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-19T02:55:09.016 INFO:teuthology.orchestra.run.smithi176.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-19T02:55:09.016 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:09.016 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-19T02:55:09.016 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================= 2024-03-19T02:55:09.016 INFO:teuthology.orchestra.run.smithi176.stdout:Install 6 Packages 2024-03-19T02:55:09.016 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:09.016 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 6.8 M 2024-03-19T02:55:09.017 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 33 M 2024-03-19T02:55:09.017 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-19T02:55:09.060 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:17.2.7-700.gb4843bd7.el9.x86_64 2/26 2024-03-19T02:55:09.151 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rados-2:17.2.7-700.gb4843bd7.el9.x86_64 3/26 2024-03-19T02:55:09.193 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs2-2:17.2.7-700.gb4843bd7.el9.x86_64 4/26 2024-03-19T02:55:09.282 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs2-2:17.2.7-700.gb4843bd7.el9.x86_64 4/26 2024-03-19T02:55:09.336 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-03-19T02:55:09.373 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-03-19T02:55:09.838 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-03-19T02:55:09.863 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librgw2-2:17.2.7-700.gb4843bd7.el9.x86_64 8/26 2024-03-19T02:55:09.917 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librgw2-2:17.2.7-700.gb4843bd7.el9.x86_64 8/26 2024-03-19T02:55:09.945 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-03-19T02:55:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-argparse-2:17.2.7-700.gb4843bd7.el9.x 10/26 2024-03-19T02:55:09.990 INFO:teuthology.orchestra.run.smithi176.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-19T02:55:10.025 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cephfs-2:17.2.7-700.gb4843bd7.el9.x86_64 11/26 2024-03-19T02:55:10.090 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rgw-2:17.2.7-700.gb4843bd7.el9.x86_64 12/26 2024-03-19T02:55:10.115 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libradosstriper1-2:17.2.7-700.gb4843bd7.el9.x86_64 13/26 2024-03-19T02:55:10.115 INFO:teuthology.orchestra.run.smithi176.stdout:(2/6): ceph-mgr-cephadm-17.2.7-700.gb4843bd7.el 285 kB/s | 123 kB 00:00 2024-03-19T02:55:10.164 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libradosstriper1-2:17.2.7-700.gb4843bd7.el9.x86_64 13/26 2024-03-19T02:55:10.205 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-03-19T02:55:10.240 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-03-19T02:55:10.292 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-03-19T02:55:10.319 INFO:teuthology.orchestra.run.smithi176.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 390 kB/s | 248 kB 00:00 2024-03-19T02:55:10.416 INFO:teuthology.orchestra.run.smithi176.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 192 kB/s | 58 kB 00:00 2024-03-19T02:55:10.566 INFO:teuthology.orchestra.run.smithi176.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 207 kB/s | 51 kB 00:00 2024-03-19T02:55:10.600 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-03-19T02:55:10.627 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librbd1-2:17.2.7-700.gb4843bd7.el9.x86_64 18/26 2024-03-19T02:55:10.718 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:17.2.7-700.gb4843bd7.el9.x86_64 18/26 2024-03-19T02:55:10.793 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rbd-2:17.2.7-700.gb4843bd7.el9.x86_64 19/26 2024-03-19T02:55:10.821 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-common-2:17.2.7-700.gb4843bd7.el9.x86 20/26 2024-03-19T02:55:13.320 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:17.2.7-700.gb4843bd7.el9.x86_64 21/26 2024-03-19T02:55:13.351 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-common-2:17.2.7-700.gb4843bd7.el9.x86_64 21/26 2024-03-19T02:55:13.903 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:17.2.7-700.gb4843bd7.el9.x86_64 21/26 2024-03-19T02:55:13.937 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-base-2:17.2.7-700.gb4843bd7.el9.x86_64 22/26 2024-03-19T02:55:14.026 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-base-2:17.2.7-700.gb4843bd7.el9.x86_64 22/26 2024-03-19T02:55:14.026 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-19T02:55:14.026 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-19T02:55:14.026 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T02:55:14.063 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-selinux-2:17.2.7-700.gb4843bd7.el9.x86_64 23/26 2024-03-19T02:55:14.125 INFO:teuthology.orchestra.run.smithi176.stdout:(6/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 132 kB/s | 548 kB 00:04 2024-03-19T02:55:14.128 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:55:14.128 INFO:teuthology.orchestra.run.smithi176.stdout:Total 1.3 MB/s | 6.8 MB 00:05 2024-03-19T02:55:14.143 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-19T02:55:14.154 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-19T02:55:14.154 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-19T02:55:14.246 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-19T02:55:14.247 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-19T02:55:14.550 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-19T02:55:14.730 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-19T02:55:14.793 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-19T02:55:15.303 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-19T02:55:15.386 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-19T02:55:15.441 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-19T02:55:15.472 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-700.gb4843bd7.el9.noarch 6/6 2024-03-19T02:55:16.267 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-700.gb4843bd7.el9.noarch 6/6 2024-03-19T02:55:16.268 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-700.gb4843bd7.el9.noarch 1/6 2024-03-19T02:55:16.268 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-19T02:55:16.268 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-19T02:55:16.268 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-19T02:55:16.268 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-19T02:55:16.587 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-19T02:55:16.587 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:16.587 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-19T02:55:16.587 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm-2:17.2.7-700.gb4843bd7.el9.noarch 2024-03-19T02:55:16.587 INFO:teuthology.orchestra.run.smithi176.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-19T02:55:16.587 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-19T02:55:16.587 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-19T02:55:16.588 INFO:teuthology.orchestra.run.smithi176.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-19T02:55:16.588 INFO:teuthology.orchestra.run.smithi176.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-19T02:55:16.588 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:16.588 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:55:16.885 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-fuse 2024-03-19T02:55:17.445 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:35 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:55:17.595 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:55:17.595 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:55:17.596 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-19T02:55:17.596 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:55:17.596 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-19T02:55:17.596 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 812 k 2024-03-19T02:55:17.596 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-19T02:55:17.596 INFO:teuthology.orchestra.run.smithi176.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-19T02:55:17.596 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:17.596 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-19T02:55:17.596 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:55:17.596 INFO:teuthology.orchestra.run.smithi176.stdout:Install 2 Packages 2024-03-19T02:55:17.596 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:17.597 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 890 k 2024-03-19T02:55:17.597 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 2.6 M 2024-03-19T02:55:17.597 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-19T02:55:17.718 INFO:teuthology.orchestra.run.smithi176.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 653 kB/s | 79 kB 00:00 2024-03-19T02:55:18.001 INFO:teuthology.orchestra.run.smithi176.stdout:(2/2): ceph-fuse-17.2.7-700.gb4843bd7.el9.x86_6 2.0 MB/s | 812 kB 00:00 2024-03-19T02:55:18.002 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:55:18.003 INFO:teuthology.orchestra.run.smithi176.stdout:Total 2.1 MB/s | 890 kB 00:00 2024-03-19T02:55:18.003 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-19T02:55:18.023 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-19T02:55:18.023 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-19T02:55:18.094 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-19T02:55:18.094 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-19T02:55:18.270 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-19T02:55:18.390 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-19T02:55:18.458 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-fuse-2:17.2.7-700.gb4843bd7.el9.x86_64 2/2 2024-03-19T02:55:19.128 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-fuse-2:17.2.7-700.gb4843bd7.el9.x86_64 2/2 2024-03-19T02:55:19.128 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-fuse-2:17.2.7-700.gb4843bd7.el9.x86_64 1/2 2024-03-19T02:55:19.437 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-19T02:55:19.437 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:19.437 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-19T02:55:19.437 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse-2:17.2.7-700.gb4843bd7.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-19T02:55:19.437 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:19.437 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:55:19.711 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librados-devel 2024-03-19T02:55:20.286 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:38 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:55:20.438 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:55:20.439 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:55:20.439 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-19T02:55:20.439 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:55:20.439 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-19T02:55:20.440 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 126 k 2024-03-19T02:55:20.440 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:20.440 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-19T02:55:20.440 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:55:20.440 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-19T02:55:20.440 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:20.440 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 126 k 2024-03-19T02:55:20.440 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 448 k 2024-03-19T02:55:20.440 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-19T02:55:20.651 INFO:teuthology.orchestra.run.smithi176.stdout:librados-devel-17.2.7-700.gb4843bd7.el9.x86_64. 599 kB/s | 126 kB 00:00 2024-03-19T02:55:20.652 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:55:20.652 INFO:teuthology.orchestra.run.smithi176.stdout:Total 594 kB/s | 126 kB 00:00 2024-03-19T02:55:20.652 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-19T02:55:20.657 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-19T02:55:20.657 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-19T02:55:20.712 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-19T02:55:20.712 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-19T02:55:20.853 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-19T02:55:20.961 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librados-devel-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T02:55:21.385 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados-devel-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T02:55:21.677 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados-devel-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T02:55:21.677 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:21.677 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-19T02:55:21.677 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:55:21.678 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:21.678 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:55:21.916 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install libcephfs2 2024-03-19T02:55:22.487 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:40 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:55:22.589 INFO:teuthology.orchestra.run.smithi176.stdout:Package libcephfs2-2:17.2.7-700.gb4843bd7.el9.x86_64 is already installed. 2024-03-19T02:55:22.644 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:55:22.645 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-19T02:55:22.645 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:55:22.708 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install libcephfs-devel 2024-03-19T02:55:23.268 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:41 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:55:23.417 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:55:23.418 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:55:23.418 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-19T02:55:23.418 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:55:23.418 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-19T02:55:23.418 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 25 k 2024-03-19T02:55:23.418 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:23.418 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-19T02:55:23.418 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:55:23.418 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-19T02:55:23.419 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:23.419 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 25 k 2024-03-19T02:55:23.419 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 110 k 2024-03-19T02:55:23.419 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-19T02:55:23.576 INFO:teuthology.orchestra.run.smithi176.stdout:libcephfs-devel-17.2.7-700.gb4843bd7.el9.x86_64 159 kB/s | 25 kB 00:00 2024-03-19T02:55:23.576 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:55:23.577 INFO:teuthology.orchestra.run.smithi176.stdout:Total 158 kB/s | 25 kB 00:00 2024-03-19T02:55:23.577 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-19T02:55:23.580 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-19T02:55:23.580 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-19T02:55:23.581 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-19T02:55:23.582 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-19T02:55:23.666 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-19T02:55:23.774 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs-devel-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T02:55:23.851 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-selinux-2:17.2.7-700.gb4843bd7.el9.x86_64 23/26 2024-03-19T02:55:23.852 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /sys 2024-03-19T02:55:23.852 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /proc 2024-03-19T02:55:23.852 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /mnt 2024-03-19T02:55:23.852 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /var/tmp 2024-03-19T02:55:23.852 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /home 2024-03-19T02:55:23.852 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /root 2024-03-19T02:55:23.852 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /tmp 2024-03-19T02:55:23.852 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T02:55:24.142 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T02:55:24.401 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs-devel-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T02:55:24.401 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:24.401 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-19T02:55:24.401 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:55:24.401 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:24.401 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:55:24.598 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librados2 2024-03-19T02:55:25.162 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:43 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:55:25.183 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-radosgw-2:17.2.7-700.gb4843bd7.el9.x86_64 24/26 2024-03-19T02:55:25.261 INFO:teuthology.orchestra.run.smithi176.stdout:Package librados2-2:17.2.7-700.gb4843bd7.el9.x86_64 is already installed. 2024-03-19T02:55:25.261 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-700.gb4843bd7.el9.x86_64 24/26 2024-03-19T02:55:25.261 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:55:25.261 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-19T02:55:25.262 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T02:55:25.262 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T02:55:25.262 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T02:55:25.263 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-03-19T02:55:25.314 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:55:25.314 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-19T02:55:25.314 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:55:25.344 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-03-19T02:55:25.344 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-03-19T02:55:25.377 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librbd1 2024-03-19T02:55:25.936 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:43 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:55:26.034 INFO:teuthology.orchestra.run.smithi176.stdout:Package librbd1-2:17.2.7-700.gb4843bd7.el9.x86_64 is already installed. 2024-03-19T02:55:26.085 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:55:26.086 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-19T02:55:26.086 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:55:26.149 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rados 2024-03-19T02:55:26.553 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-03-19T02:55:26.554 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-base-2:17.2.7-700.gb4843bd7.el9.x86_64 1/26 2024-03-19T02:55:26.554 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-common-2:17.2.7-700.gb4843bd7.el9.x86_64 2/26 2024-03-19T02:55:26.554 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-radosgw-2:17.2.7-700.gb4843bd7.el9.x86_64 3/26 2024-03-19T02:55:26.554 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-selinux-2:17.2.7-700.gb4843bd7.el9.x86_64 4/26 2024-03-19T02:55:26.554 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs2-2:17.2.7-700.gb4843bd7.el9.x86_64 5/26 2024-03-19T02:55:26.554 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libradosstriper1-2:17.2.7-700.gb4843bd7.el9.x86_64 6/26 2024-03-19T02:55:26.554 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librgw2-2:17.2.7-700.gb4843bd7.el9.x86_64 7/26 2024-03-19T02:55:26.554 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-argparse-2:17.2.7-700.gb4843bd7.el9.x 8/26 2024-03-19T02:55:26.554 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-common-2:17.2.7-700.gb4843bd7.el9.x86 9/26 2024-03-19T02:55:26.554 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cephfs-2:17.2.7-700.gb4843bd7.el9.x86_64 10/26 2024-03-19T02:55:26.554 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rados-2:17.2.7-700.gb4843bd7.el9.x86_64 11/26 2024-03-19T02:55:26.554 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rbd-2:17.2.7-700.gb4843bd7.el9.x86_64 12/26 2024-03-19T02:55:26.554 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rgw-2:17.2.7-700.gb4843bd7.el9.x86_64 13/26 2024-03-19T02:55:26.555 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-03-19T02:55:26.555 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-03-19T02:55:26.555 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-03-19T02:55:26.555 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-03-19T02:55:26.555 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-03-19T02:55:26.555 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-03-19T02:55:26.555 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-03-19T02:55:26.555 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-03-19T02:55:26.555 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-03-19T02:55:26.555 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:17.2.7-700.gb4843bd7.el9.x86_64 23/26 2024-03-19T02:55:26.555 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-03-19T02:55:26.555 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:17.2.7-700.gb4843bd7.el9.x86_64 25/26 2024-03-19T02:55:26.712 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:44 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:55:26.809 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rados-2:17.2.7-700.gb4843bd7.el9.x86_64 is already installed. 2024-03-19T02:55:26.863 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:55:26.863 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-19T02:55:26.864 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:55:26.928 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rgw 2024-03-19T02:55:27.427 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-03-19T02:55:27.427 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T02:55:27.427 INFO:teuthology.orchestra.run.smithi059.stdout:Upgraded: 2024-03-19T02:55:27.427 INFO:teuthology.orchestra.run.smithi059.stdout: librados2-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:55:27.427 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:55:27.427 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-19T02:55:27.427 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:55:27.427 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:55:27.428 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:55:27.428 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:55:27.428 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-19T02:55:27.428 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:55:27.428 INFO:teuthology.orchestra.run.smithi059.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-19T02:55:27.428 INFO:teuthology.orchestra.run.smithi059.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-19T02:55:27.428 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-19T02:55:27.428 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:55:27.428 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-19T02:55:27.428 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:55:27.428 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-19T02:55:27.428 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-19T02:55:27.428 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:55:27.428 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:55:27.428 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:55:27.429 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-19T02:55:27.429 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:55:27.429 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:55:27.429 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:55:27.429 INFO:teuthology.orchestra.run.smithi059.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-19T02:55:27.429 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T02:55:27.429 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T02:55:27.501 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:45 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:55:27.598 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rgw-2:17.2.7-700.gb4843bd7.el9.x86_64 is already installed. 2024-03-19T02:55:27.651 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:55:27.652 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-19T02:55:27.652 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:55:27.719 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-cephfs 2024-03-19T02:55:27.761 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-test 2024-03-19T02:55:28.278 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:46 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:55:28.326 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:00:34 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T02:55:28.375 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-cephfs-2:17.2.7-700.gb4843bd7.el9.x86_64 is already installed. 2024-03-19T02:55:28.428 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:55:28.429 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-19T02:55:28.429 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:55:28.487 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T02:55:28.488 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T02:55:28.488 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-03-19T02:55:28.488 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T02:55:28.488 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-19T02:55:28.488 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 48 M 2024-03-19T02:55:28.488 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-03-19T02:55:28.488 INFO:teuthology.orchestra.run.smithi059.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-19T02:55:28.488 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 163 k 2024-03-19T02:55:28.488 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-19T02:55:28.488 INFO:teuthology.orchestra.run.smithi059.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-19T02:55:28.489 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-19T02:55:28.489 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T02:55:28.489 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-19T02:55:28.489 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T02:55:28.489 INFO:teuthology.orchestra.run.smithi059.stdout:Install 6 Packages 2024-03-19T02:55:28.489 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T02:55:28.490 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 49 M 2024-03-19T02:55:28.490 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 196 M 2024-03-19T02:55:28.490 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-19T02:55:28.491 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rbd 2024-03-19T02:55:28.931 INFO:teuthology.orchestra.run.smithi059.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-19T02:55:28.998 INFO:teuthology.orchestra.run.smithi059.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2024-03-19T02:55:29.023 INFO:teuthology.orchestra.run.smithi059.stdout:(3/6): libcephsqlite-17.2.7-700.gb4843bd7.el9.x 641 kB/s | 163 kB 00:00 2024-03-19T02:55:29.048 INFO:teuthology.orchestra.run.smithi059.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-19T02:55:29.053 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:47 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:55:29.152 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rbd-2:17.2.7-700.gb4843bd7.el9.x86_64 is already installed. 2024-03-19T02:55:29.204 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:55:29.205 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-19T02:55:29.205 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:55:29.265 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-fuse 2024-03-19T02:55:29.323 INFO:teuthology.orchestra.run.smithi059.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 624 kB/s | 187 kB 00:00 2024-03-19T02:55:29.833 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:47 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:55:29.984 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:55:29.985 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:55:29.985 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-03-19T02:55:29.985 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:55:29.985 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-19T02:55:29.985 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 82 k 2024-03-19T02:55:29.985 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:29.985 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-19T02:55:29.985 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:55:29.986 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-19T02:55:29.986 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:29.986 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 82 k 2024-03-19T02:55:29.986 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 222 k 2024-03-19T02:55:29.986 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-19T02:55:30.495 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-fuse-17.2.7-700.gb4843bd7.el9.x86_64.rpm 161 kB/s | 82 kB 00:00 2024-03-19T02:55:30.496 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:55:30.496 INFO:teuthology.orchestra.run.smithi176.stdout:Total 160 kB/s | 82 kB 00:00 2024-03-19T02:55:30.496 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-19T02:55:30.503 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-19T02:55:30.503 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-19T02:55:30.563 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-19T02:55:30.564 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-19T02:55:30.699 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-19T02:55:30.810 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-fuse-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T02:55:31.157 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-fuse-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T02:55:31.282 INFO:teuthology.orchestra.run.smithi059.stdout:(6/6): ceph-test-17.2.7-700.gb4843bd7.el9.x86_6 19 MB/s | 48 MB 00:02 2024-03-19T02:55:31.285 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:55:31.285 INFO:teuthology.orchestra.run.smithi059.stdout:Total 17 MB/s | 49 MB 00:02 2024-03-19T02:55:31.299 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-19T02:55:31.320 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-19T02:55:31.320 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-19T02:55:31.450 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-fuse-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T02:55:31.450 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:31.451 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-19T02:55:31.451 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:55:31.451 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:31.451 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:55:31.692 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-mirror 2024-03-19T02:55:31.730 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-19T02:55:31.731 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-19T02:55:32.289 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:50 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:55:32.320 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-19T02:55:32.400 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-19T02:55:32.441 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-19T02:55:32.447 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:55:32.448 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:55:32.448 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-19T02:55:32.448 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:55:32.448 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-19T02:55:32.448 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 2.9 M 2024-03-19T02:55:32.448 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:32.449 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-19T02:55:32.449 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:55:32.449 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-19T02:55:32.449 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:32.449 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 2.9 M 2024-03-19T02:55:32.449 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 12 M 2024-03-19T02:55:32.449 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-19T02:55:32.473 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-19T02:55:32.533 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-19T02:55:32.560 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephsqlite-2:17.2.7-700.gb4843bd7.el9.x86_64 5/6 2024-03-19T02:55:32.990 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-mirror-17.2.7-700.gb4843bd7.el9.x86_64.rpm 5.4 MB/s | 2.9 MB 00:00 2024-03-19T02:55:32.991 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:55:32.992 INFO:teuthology.orchestra.run.smithi176.stdout:Total 5.4 MB/s | 2.9 MB 00:00 2024-03-19T02:55:32.992 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-19T02:55:33.000 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-19T02:55:33.001 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-19T02:55:33.071 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-19T02:55:33.072 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-19T02:55:33.481 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-19T02:55:33.543 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-mirror-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T02:55:33.568 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-mirror-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T02:55:33.569 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:55:33.569 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-19T02:55:33.569 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T02:55:33.569 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T02:55:33.569 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:34.633 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-mirror-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T02:55:34.633 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:34.633 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-19T02:55:34.633 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:55:34.633 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:34.633 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:55:34.879 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-nbd 2024-03-19T02:55:35.462 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:53 ago on Tue 19 Mar 2024 02:51:42 AM UTC. 2024-03-19T02:55:35.615 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-19T02:55:35.616 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:55:35.616 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-03-19T02:55:35.616 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:55:35.616 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-19T02:55:35.616 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 165 k 2024-03-19T02:55:35.616 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:35.617 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-19T02:55:35.617 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-19T02:55:35.617 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-19T02:55:35.617 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:35.617 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 165 k 2024-03-19T02:55:35.617 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 477 k 2024-03-19T02:55:35.617 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-19T02:55:35.845 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-nbd-17.2.7-700.gb4843bd7.el9.x86_64.rpm 724 kB/s | 165 kB 00:00 2024-03-19T02:55:35.846 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:55:35.846 INFO:teuthology.orchestra.run.smithi176.stdout:Total 719 kB/s | 165 kB 00:00 2024-03-19T02:55:35.846 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-19T02:55:35.854 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-19T02:55:35.854 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-19T02:55:35.908 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-19T02:55:35.908 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-19T02:55:36.055 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-19T02:55:36.152 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-nbd-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T02:55:36.633 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-nbd-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T02:55:36.925 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-nbd-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T02:55:36.925 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:36.925 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-19T02:55:36.925 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:55:36.925 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T02:55:36.925 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-19T02:55:37.157 DEBUG:teuthology.parallel:result is None 2024-03-19T02:55:37.243 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephsqlite-2:17.2.7-700.gb4843bd7.el9.x86_64 5/6 2024-03-19T02:55:37.330 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-test-2:17.2.7-700.gb4843bd7.el9.x86_64 6/6 2024-03-19T02:55:39.649 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-test-2:17.2.7-700.gb4843bd7.el9.x86_64 6/6 2024-03-19T02:55:39.649 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-test-2:17.2.7-700.gb4843bd7.el9.x86_64 1/6 2024-03-19T02:55:39.649 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephsqlite-2:17.2.7-700.gb4843bd7.el9.x86_64 2/6 2024-03-19T02:55:39.649 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-19T02:55:39.649 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-19T02:55:39.650 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-19T02:55:40.041 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-19T02:55:40.041 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T02:55:40.041 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-19T02:55:40.041 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:55:40.041 INFO:teuthology.orchestra.run.smithi059.stdout: jq-1.6-16.el9.x86_64 2024-03-19T02:55:40.041 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T02:55:40.041 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-19T02:55:40.041 INFO:teuthology.orchestra.run.smithi059.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-19T02:55:40.041 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-19T02:55:40.041 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T02:55:40.042 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T02:55:40.304 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph 2024-03-19T02:55:40.871 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:00:46 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T02:55:41.039 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T02:55:41.041 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================== 2024-03-19T02:55:41.041 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-03-19T02:55:41.041 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================== 2024-03-19T02:55:41.042 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-19T02:55:41.042 INFO:teuthology.orchestra.run.smithi059.stdout: ceph x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 6.5 k 2024-03-19T02:55:41.042 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-03-19T02:55:41.042 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 2.1 M 2024-03-19T02:55:41.042 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 1.4 M 2024-03-19T02:55:41.042 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core noarch 2:17.2.7-700.gb4843bd7.el9 ceph-noarch 239 k 2024-03-19T02:55:41.042 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 3.7 M 2024-03-19T02:55:41.042 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 15 M 2024-03-19T02:55:41.042 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-19T02:55:41.042 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-19T02:55:41.042 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-19T02:55:41.042 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-19T02:55:41.043 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-19T02:55:41.043 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-19T02:55:41.043 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-19T02:55:41.043 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-19T02:55:41.043 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-19T02:55:41.043 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-19T02:55:41.043 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-19T02:55:41.043 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-19T02:55:41.043 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-19T02:55:41.043 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-19T02:55:41.043 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-19T02:55:41.043 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-19T02:55:41.044 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-19T02:55:41.044 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-19T02:55:41.044 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-19T02:55:41.044 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-19T02:55:41.044 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-19T02:55:41.044 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-19T02:55:41.044 INFO:teuthology.orchestra.run.smithi059.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-19T02:55:41.044 INFO:teuthology.orchestra.run.smithi059.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-19T02:55:41.044 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-19T02:55:41.044 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-19T02:55:41.044 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-19T02:55:41.044 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T02:55:41.044 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-19T02:55:41.044 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================== 2024-03-19T02:55:41.045 INFO:teuthology.orchestra.run.smithi059.stdout:Install 33 Packages 2024-03-19T02:55:41.045 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T02:55:41.045 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 27 M 2024-03-19T02:55:41.045 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 100 M 2024-03-19T02:55:41.045 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-19T02:55:41.485 INFO:teuthology.orchestra.run.smithi059.stdout:(1/33): ceph-17.2.7-700.gb4843bd7.el9.x86_64.rp 19 kB/s | 6.5 kB 00:00 2024-03-19T02:55:41.636 INFO:teuthology.orchestra.run.smithi059.stdout:(2/33): ceph-mds-17.2.7-700.gb4843bd7.el9.x86_6 4.2 MB/s | 2.1 MB 00:00 2024-03-19T02:55:41.944 INFO:teuthology.orchestra.run.smithi059.stdout:(3/33): ceph-mgr-17.2.7-700.gb4843bd7.el9.x86_6 1.8 MB/s | 1.4 MB 00:00 2024-03-19T02:55:42.069 INFO:teuthology.orchestra.run.smithi059.stdout:(4/33): ceph-mgr-modules-core-17.2.7-700.gb4843 1.9 MB/s | 239 kB 00:00 2024-03-19T02:55:42.211 INFO:teuthology.orchestra.run.smithi059.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 144 kB 00:00 2024-03-19T02:55:42.245 INFO:teuthology.orchestra.run.smithi059.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 950 kB/s | 32 kB 00:00 2024-03-19T02:55:42.303 INFO:teuthology.orchestra.run.smithi059.stdout:(7/33): ceph-mon-17.2.7-700.gb4843bd7.el9.x86_6 4.5 MB/s | 3.7 MB 00:00 2024-03-19T02:55:42.320 INFO:teuthology.orchestra.run.smithi059.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 630 kB/s | 47 kB 00:00 2024-03-19T02:55:42.337 INFO:teuthology.orchestra.run.smithi059.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-19T02:55:42.488 INFO:teuthology.orchestra.run.smithi059.stdout:(10/33): ceph-osd-17.2.7-700.gb4843bd7.el9.x86_ 18 MB/s | 15 MB 00:00 2024-03-19T02:55:42.513 INFO:teuthology.orchestra.run.smithi059.stdout:(11/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 241 kB 00:00 2024-03-19T02:55:42.663 INFO:teuthology.orchestra.run.smithi059.stdout:(12/33): python3-cryptography-36.0.1-4.el9.x86_ 3.5 MB/s | 1.1 MB 00:00 2024-03-19T02:55:42.689 INFO:teuthology.orchestra.run.smithi059.stdout:(13/33): python3-ply-3.11-14.el9.noarch.rpm 515 kB/s | 103 kB 00:00 2024-03-19T02:55:42.714 INFO:teuthology.orchestra.run.smithi059.stdout:(14/33): python3-pycparser-2.20-6.el9.noarch.rp 619 kB/s | 124 kB 00:00 2024-03-19T02:55:42.747 INFO:teuthology.orchestra.run.smithi059.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.2 MB/s | 187 kB 00:00 2024-03-19T02:55:43.148 INFO:teuthology.orchestra.run.smithi059.stdout:(16/33): python3-requests-2.25.1-8.el9.noarch.r 288 kB/s | 125 kB 00:00 2024-03-19T02:57:53.960 INFO:teuthology.orchestra.run.smithi059.stdout:[MIRROR] python3-bcrypt-3.2.2-1.el9.x86_64.rpm: Curl error (28): Timeout was reached for https://ftp-nyc.osuosl.org/pub/fedora-epel/9/Everything/x86_64/Packages/p/python3-bcrypt-3.2.2-1.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 443: Connection timed out] 2024-03-19T02:57:53.961 INFO:teuthology.orchestra.run.smithi059.stdout:[MIRROR] python3-cheroot-8.6.0-4.el9.noarch.rpm: Curl error (28): Timeout was reached for https://ftp-nyc.osuosl.org/pub/fedora-epel/9/Everything/x86_64/Packages/p/python3-cheroot-8.6.0-4.el9.noarch.rpm [Failed to connect to ftp-nyc.osuosl.org port 443: Connection timed out] 2024-03-19T02:57:53.961 INFO:teuthology.orchestra.run.smithi059.stdout:[MIRROR] python3-cherrypy-18.6.1-2.el9.noarch.rpm: Curl error (28): Timeout was reached for https://ftp-nyc.osuosl.org/pub/fedora-epel/9/Everything/x86_64/Packages/p/python3-cherrypy-18.6.1-2.el9.noarch.rpm [Failed to connect to ftp-nyc.osuosl.org port 443: Connection timed out] 2024-03-19T03:00:04.776 INFO:teuthology.orchestra.run.smithi059.stdout:[MIRROR] python3-bcrypt-3.2.2-1.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/fedora-epel/9/Everything/x86_64/Packages/p/python3-bcrypt-3.2.2-1.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T03:00:04.777 INFO:teuthology.orchestra.run.smithi059.stdout:[MIRROR] python3-cheroot-8.6.0-4.el9.noarch.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/fedora-epel/9/Everything/x86_64/Packages/p/python3-cheroot-8.6.0-4.el9.noarch.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T03:00:04.777 INFO:teuthology.orchestra.run.smithi059.stdout:[MIRROR] python3-cherrypy-18.6.1-2.el9.noarch.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/fedora-epel/9/Everything/x86_64/Packages/p/python3-cherrypy-18.6.1-2.el9.noarch.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T03:00:05.341 INFO:teuthology.orchestra.run.smithi059.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 670 B/s | 172 kB 04:22 2024-03-19T03:00:05.408 INFO:teuthology.orchestra.run.smithi059.stdout:(18/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 kB/s | 358 kB 04:22 2024-03-19T03:00:05.450 INFO:teuthology.orchestra.run.smithi059.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 98 kB/s | 11 kB 00:00 2024-03-19T03:00:05.517 INFO:teuthology.orchestra.run.smithi059.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-03-19T03:00:05.567 INFO:teuthology.orchestra.run.smithi059.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 198 kB/s | 23 kB 00:00 2024-03-19T03:00:05.626 INFO:teuthology.orchestra.run.smithi059.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 179 kB/s | 19 kB 00:00 2024-03-19T03:00:05.676 INFO:teuthology.orchestra.run.smithi059.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-03-19T03:00:05.734 INFO:teuthology.orchestra.run.smithi059.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 427 kB/s | 46 kB 00:00 2024-03-19T03:00:05.810 INFO:teuthology.orchestra.run.smithi059.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 590 kB/s | 79 kB 00:00 2024-03-19T03:00:05.885 INFO:teuthology.orchestra.run.smithi059.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-03-19T03:00:05.919 INFO:teuthology.orchestra.run.smithi059.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-03-19T03:00:06.002 INFO:teuthology.orchestra.run.smithi059.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 769 kB/s | 90 kB 00:00 2024-03-19T03:00:06.027 INFO:teuthology.orchestra.run.smithi059.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 329 kB/s | 36 kB 00:00 2024-03-19T03:00:06.144 INFO:teuthology.orchestra.run.smithi059.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.6 MB/s | 230 kB 00:00 2024-03-19T03:00:06.246 INFO:teuthology.orchestra.run.smithi059.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-03-19T03:00:06.295 INFO:teuthology.orchestra.run.smithi059.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.6 MB/s | 427 kB 00:00 2024-03-19T03:02:15.893 INFO:teuthology.orchestra.run.smithi059.stdout:[MIRROR] python3-bcrypt-3.2.2-1.el9.x86_64.rpm: Curl error (28): Timeout was reached for https://ftp-nyc.osuosl.org/pub/fedora-epel/9/Everything/x86_64/Packages/p/python3-bcrypt-3.2.2-1.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 443: Connection timed out] 2024-03-19T03:02:16.247 INFO:teuthology.orchestra.run.smithi059.stdout:(33/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 113 B/s | 43 kB 06:33 2024-03-19T03:02:16.247 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-19T03:02:16.247 INFO:teuthology.orchestra.run.smithi059.stdout:Total 69 kB/s | 27 MB 06:35 2024-03-19T03:02:16.259 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-19T03:02:16.302 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-19T03:02:16.303 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-19T03:02:16.651 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-19T03:02:16.652 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-19T03:02:17.920 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-19T03:02:17.991 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-19T03:02:18.024 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-19T03:02:18.057 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-19T03:02:18.152 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-19T03:02:18.228 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-19T03:02:18.276 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-19T03:02:18.349 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-19T03:02:18.402 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-19T03:02:18.440 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-19T03:02:18.498 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-19T03:02:18.560 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-19T03:02:18.760 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-19T03:02:18.814 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-19T03:02:18.849 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-19T03:02:18.884 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-19T03:02:18.914 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-19T03:02:18.992 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-19T03:02:19.024 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-19T03:02:19.057 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-19T03:02:19.098 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-19T03:02:19.141 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-19T03:02:19.175 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-19T03:02:19.267 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-19T03:02:19.306 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-19T03:02:19.389 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-19T03:02:19.536 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-19T03:02:19.677 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-19T03:02:19.836 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-700.gb4843bd7.el9.n 28/33 2024-03-19T03:02:19.860 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-2:17.2.7-700.gb4843bd7.el9.x86_64 29/33 2024-03-19T03:02:19.884 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-2:17.2.7-700.gb4843bd7.el9.x86_64 29/33 2024-03-19T03:02:19.884 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:02:19.884 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-19T03:02:19.885 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T03:02:19.885 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T03:02:19.885 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:02:21.286 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-osd-2:17.2.7-700.gb4843bd7.el9.x86_64 30/33 2024-03-19T03:02:21.312 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-osd-2:17.2.7-700.gb4843bd7.el9.x86_64 30/33 2024-03-19T03:02:21.312 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:02:21.312 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-19T03:02:21.312 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T03:02:21.312 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T03:02:21.313 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:02:21.712 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mon-2:17.2.7-700.gb4843bd7.el9.x86_64 31/33 2024-03-19T03:02:21.734 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mon-2:17.2.7-700.gb4843bd7.el9.x86_64 31/33 2024-03-19T03:02:21.734 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:02:21.734 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-19T03:02:21.734 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T03:02:21.734 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T03:02:21.734 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:02:21.946 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mds-2:17.2.7-700.gb4843bd7.el9.x86_64 32/33 2024-03-19T03:02:21.968 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mds-2:17.2.7-700.gb4843bd7.el9.x86_64 32/33 2024-03-19T03:02:21.968 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:02:21.968 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-19T03:02:21.969 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T03:02:21.969 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T03:02:21.969 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:02:22.037 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-2:17.2.7-700.gb4843bd7.el9.x86_64 33/33 2024-03-19T03:02:23.572 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-2:17.2.7-700.gb4843bd7.el9.x86_64 33/33 2024-03-19T03:02:23.572 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-2:17.2.7-700.gb4843bd7.el9.x86_64 1/33 2024-03-19T03:02:23.572 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mds-2:17.2.7-700.gb4843bd7.el9.x86_64 2/33 2024-03-19T03:02:23.572 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-2:17.2.7-700.gb4843bd7.el9.x86_64 3/33 2024-03-19T03:02:23.573 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mon-2:17.2.7-700.gb4843bd7.el9.x86_64 4/33 2024-03-19T03:02:23.573 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-osd-2:17.2.7-700.gb4843bd7.el9.x86_64 5/33 2024-03-19T03:02:23.573 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-700.gb4843bd7.el9.n 6/33 2024-03-19T03:02:23.573 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-19T03:02:23.573 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-19T03:02:23.573 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-19T03:02:23.573 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-19T03:02:23.573 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-19T03:02:23.573 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-19T03:02:23.573 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-19T03:02:23.573 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-19T03:02:23.573 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-19T03:02:23.573 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-19T03:02:23.574 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-19T03:02:23.574 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-19T03:02:23.574 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-19T03:02:23.574 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-19T03:02:23.574 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-19T03:02:23.574 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-19T03:02:23.574 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-19T03:02:23.574 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-19T03:02:23.574 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-19T03:02:23.575 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-19T03:02:23.575 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-19T03:02:23.575 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-19T03:02:23.575 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-19T03:02:23.575 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-19T03:02:23.576 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-19T03:02:23.576 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-19T03:02:24.478 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-19T03:02:24.478 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:02:24.478 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-19T03:02:24.478 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T03:02:24.478 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T03:02:24.478 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T03:02:24.478 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core-2:17.2.7-700.gb4843bd7.el9.noarch 2024-03-19T03:02:24.478 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T03:02:24.478 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T03:02:24.479 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-19T03:02:24.479 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-19T03:02:24.479 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-19T03:02:24.479 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-19T03:02:24.479 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-19T03:02:24.479 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-19T03:02:24.479 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-19T03:02:24.479 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-19T03:02:24.479 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-19T03:02:24.479 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-19T03:02:24.479 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-19T03:02:24.479 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-19T03:02:24.479 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-19T03:02:24.479 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-19T03:02:24.480 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-19T03:02:24.480 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ply-3.11-14.el9.noarch 2024-03-19T03:02:24.480 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-19T03:02:24.480 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-19T03:02:24.480 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-19T03:02:24.480 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-19T03:02:24.480 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-19T03:02:24.480 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-19T03:02:24.480 INFO:teuthology.orchestra.run.smithi059.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-19T03:02:24.480 INFO:teuthology.orchestra.run.smithi059.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-19T03:02:24.480 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-19T03:02:24.480 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-19T03:02:24.480 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-19T03:02:24.480 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:02:24.481 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T03:02:24.786 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-base 2024-03-19T03:02:25.364 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:07:31 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T03:02:25.462 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-base-2:17.2.7-700.gb4843bd7.el9.x86_64 is already installed. 2024-03-19T03:02:25.516 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T03:02:25.517 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-03-19T03:02:25.517 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T03:02:25.580 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install cephadm 2024-03-19T03:02:26.148 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:07:32 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T03:02:26.301 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T03:02:26.302 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:02:26.302 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-03-19T03:02:26.302 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:02:26.302 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-19T03:02:26.302 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm noarch 2:17.2.7-700.gb4843bd7.el9 ceph-noarch 82 k 2024-03-19T03:02:26.303 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:02:26.303 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-19T03:02:26.303 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:02:26.303 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-03-19T03:02:26.303 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:02:26.303 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 82 k 2024-03-19T03:02:26.303 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 372 k 2024-03-19T03:02:26.303 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-19T03:02:26.517 INFO:teuthology.orchestra.run.smithi059.stdout:cephadm-17.2.7-700.gb4843bd7.el9.noarch.rpm 384 kB/s | 82 kB 00:00 2024-03-19T03:02:26.517 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-19T03:02:26.517 INFO:teuthology.orchestra.run.smithi059.stdout:Total 382 kB/s | 82 kB 00:00 2024-03-19T03:02:26.518 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-19T03:02:26.523 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-19T03:02:26.523 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-19T03:02:26.525 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-19T03:02:26.525 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-19T03:02:26.600 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-19T03:02:26.955 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:17.2.7-700.gb4843bd7.el9.noarch 1/1 2024-03-19T03:02:27.056 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephadm-2:17.2.7-700.gb4843bd7.el9.noarch 1/1 2024-03-19T03:02:27.440 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:17.2.7-700.gb4843bd7.el9.noarch 1/1 2024-03-19T03:02:27.703 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephadm-2:17.2.7-700.gb4843bd7.el9.noarch 1/1 2024-03-19T03:02:27.703 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:02:27.703 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-19T03:02:27.703 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm-2:17.2.7-700.gb4843bd7.el9.noarch 2024-03-19T03:02:27.703 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:02:27.703 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T03:02:27.938 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-immutable-object-cache 2024-03-19T03:02:28.514 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:07:34 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T03:02:28.670 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T03:02:28.671 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:02:28.671 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-03-19T03:02:28.671 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:02:28.671 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-19T03:02:28.671 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 137 k 2024-03-19T03:02:28.671 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:02:28.671 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-19T03:02:28.671 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:02:28.671 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-03-19T03:02:28.671 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:02:28.672 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 137 k 2024-03-19T03:02:28.672 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 379 k 2024-03-19T03:02:28.672 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-19T03:02:29.022 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-immutable-object-cache-17.2.7-700.gb4843bd 392 kB/s | 137 kB 00:00 2024-03-19T03:02:29.023 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-19T03:02:29.023 INFO:teuthology.orchestra.run.smithi059.stdout:Total 391 kB/s | 137 kB 00:00 2024-03-19T03:02:29.023 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-19T03:02:29.031 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-19T03:02:29.031 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-19T03:02:29.085 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-19T03:02:29.086 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-19T03:02:29.237 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-19T03:02:29.306 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-700.gb4843bd7.e 1/1 2024-03-19T03:02:29.331 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-700.gb4843bd7.e 1/1 2024-03-19T03:02:29.331 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:02:29.331 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-19T03:02:29.331 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:02:30.456 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-700.gb4843bd7.e 1/1 2024-03-19T03:02:30.456 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:02:30.456 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-19T03:02:30.456 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T03:02:30.456 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:02:30.457 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T03:02:30.683 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr 2024-03-19T03:02:31.260 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:07:37 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T03:02:31.358 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-mgr-2:17.2.7-700.gb4843bd7.el9.x86_64 is already installed. 2024-03-19T03:02:31.413 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T03:02:31.414 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-03-19T03:02:31.414 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T03:02:31.469 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-dashboard 2024-03-19T03:02:32.037 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:07:38 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T03:02:32.192 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T03:02:32.192 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:02:32.193 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-03-19T03:02:32.193 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:02:32.193 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-19T03:02:32.193 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard noarch 2:17.2.7-700.gb4843bd7.el9 ceph-noarch 1.7 M 2024-03-19T03:02:32.193 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-03-19T03:02:32.193 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards noarch 2:17.2.7-700.gb4843bd7.el9 ceph-noarch 23 k 2024-03-19T03:02:32.193 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts noarch 2:17.2.7-700.gb4843bd7.el9 ceph-noarch 15 k 2024-03-19T03:02:32.193 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-03-19T03:02:32.193 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-19T03:02:32.193 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-19T03:02:32.193 INFO:teuthology.orchestra.run.smithi059.stdout:Installing weak dependencies: 2024-03-19T03:02:32.193 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-03-19T03:02:32.193 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:02:32.194 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-19T03:02:32.194 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:02:32.194 INFO:teuthology.orchestra.run.smithi059.stdout:Install 7 Packages 2024-03-19T03:02:32.194 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:02:32.194 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 2.0 M 2024-03-19T03:02:32.194 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 8.1 M 2024-03-19T03:02:32.195 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-19T03:02:32.475 INFO:teuthology.orchestra.run.smithi059.stdout:(1/7): ceph-grafana-dashboards-17.2.7-700.gb484 119 kB/s | 23 kB 00:00 2024-03-19T03:02:32.492 INFO:teuthology.orchestra.run.smithi059.stdout:(2/7): ceph-prometheus-alerts-17.2.7-700.gb4843 68 kB/s | 15 kB 00:00 2024-03-19T03:02:32.642 INFO:teuthology.orchestra.run.smithi059.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 54 kB/s | 9.0 kB 00:00 2024-03-19T03:02:32.684 INFO:teuthology.orchestra.run.smithi059.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 213 kB/s | 41 kB 00:00 2024-03-19T03:02:32.701 INFO:teuthology.orchestra.run.smithi059.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 525 kB/s | 31 kB 00:00 2024-03-19T03:02:32.809 INFO:teuthology.orchestra.run.smithi059.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-03-19T03:02:33.010 INFO:teuthology.orchestra.run.smithi059.stdout:(7/7): ceph-mgr-dashboard-17.2.7-700.gb4843bd7. 2.3 MB/s | 1.7 MB 00:00 2024-03-19T03:02:33.013 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-19T03:02:33.013 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-03-19T03:02:33.014 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-19T03:02:33.022 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-19T03:02:33.022 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-19T03:02:33.082 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-19T03:02:33.083 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-19T03:02:33.387 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-19T03:02:33.444 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-03-19T03:02:33.479 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-03-19T03:02:33.534 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-03-19T03:02:33.565 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-03-19T03:02:33.585 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-700.gb4843bd7.el9.no 5/7 2024-03-19T03:02:33.818 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-700.gb4843bd7.el9.n 6/7 2024-03-19T03:02:33.855 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-700.gb4843bd7.el9.noarch 7/7 2024-03-19T03:02:34.378 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-700.gb4843bd7.el9.noarch 7/7 2024-03-19T03:02:34.379 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-700.gb4843bd7.el9.n 1/7 2024-03-19T03:02:34.379 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-700.gb4843bd7.el9.noarch 2/7 2024-03-19T03:02:34.379 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-700.gb4843bd7.el9.no 3/7 2024-03-19T03:02:34.379 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-03-19T03:02:34.379 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-03-19T03:02:34.379 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-03-19T03:02:34.803 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-03-19T03:02:34.803 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:02:34.803 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-19T03:02:34.803 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards-2:17.2.7-700.gb4843bd7.el9.noarch 2024-03-19T03:02:34.804 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard-2:17.2.7-700.gb4843bd7.el9.noarch 2024-03-19T03:02:34.804 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts-2:17.2.7-700.gb4843bd7.el9.noarch 2024-03-19T03:02:34.804 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-03-19T03:02:34.804 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-03-19T03:02:34.804 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-19T03:02:34.804 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-19T03:02:34.805 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:02:34.805 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T03:02:35.007 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-19T03:02:35.593 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:07:41 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T03:02:35.758 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T03:02:35.759 INFO:teuthology.orchestra.run.smithi059.stdout:=========================================================================================== 2024-03-19T03:02:35.759 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-03-19T03:02:35.759 INFO:teuthology.orchestra.run.smithi059.stdout:=========================================================================================== 2024-03-19T03:02:35.759 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-19T03:02:35.759 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-700.gb4843bd7.el9 ceph-noarch 7.4 M 2024-03-19T03:02:35.760 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-03-19T03:02:35.760 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-19T03:02:35.760 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-19T03:02:35.760 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-19T03:02:35.760 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-19T03:02:35.760 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-19T03:02:35.760 INFO:teuthology.orchestra.run.smithi059.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-19T03:02:35.760 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-19T03:02:35.760 INFO:teuthology.orchestra.run.smithi059.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-19T03:02:35.760 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-19T03:02:35.760 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-19T03:02:35.760 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-19T03:02:35.760 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:02:35.760 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-19T03:02:35.761 INFO:teuthology.orchestra.run.smithi059.stdout:=========================================================================================== 2024-03-19T03:02:35.761 INFO:teuthology.orchestra.run.smithi059.stdout:Install 12 Packages 2024-03-19T03:02:35.761 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:02:35.761 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 37 M 2024-03-19T03:02:35.761 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 209 M 2024-03-19T03:02:35.761 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-19T03:02:36.200 INFO:teuthology.orchestra.run.smithi059.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 222 kB/s | 32 kB 00:00 2024-03-19T03:02:36.225 INFO:teuthology.orchestra.run.smithi059.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 733 kB/s | 17 kB 00:00 2024-03-19T03:02:36.250 INFO:teuthology.orchestra.run.smithi059.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.4 MB/s | 34 kB 00:00 2024-03-19T03:02:36.309 INFO:teuthology.orchestra.run.smithi059.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-19T03:02:36.484 INFO:teuthology.orchestra.run.smithi059.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-19T03:02:36.551 INFO:teuthology.orchestra.run.smithi059.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-19T03:02:36.576 INFO:teuthology.orchestra.run.smithi059.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-19T03:02:36.877 INFO:teuthology.orchestra.run.smithi059.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-03-19T03:02:37.168 INFO:teuthology.orchestra.run.smithi059.stdout:(9/12): ceph-mgr-diskprediction-local-17.2.7-70 6.7 MB/s | 7.4 MB 00:01 2024-03-19T03:02:37.369 INFO:teuthology.orchestra.run.smithi059.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-03-19T03:04:47.398 INFO:teuthology.orchestra.run.smithi059.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T03:04:47.399 INFO:teuthology.orchestra.run.smithi059.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T03:04:48.112 INFO:teuthology.orchestra.run.smithi059.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.5 kB/s | 195 kB 02:11 2024-03-19T03:04:48.778 INFO:teuthology.orchestra.run.smithi059.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 6.1 kB/s | 803 kB 02:12 2024-03-19T03:04:48.779 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-19T03:04:48.779 INFO:teuthology.orchestra.run.smithi059.stdout:Total 285 kB/s | 37 MB 02:13 2024-03-19T03:04:48.818 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-19T03:04:48.846 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-19T03:04:48.846 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-19T03:04:49.282 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-19T03:04:49.282 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-19T03:04:50.012 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-19T03:04:50.093 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-19T03:04:50.124 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-19T03:04:50.226 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-19T03:04:50.253 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-19T03:04:50.448 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-19T03:04:50.477 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-19T03:04:50.556 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-19T03:04:51.250 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-19T03:04:51.377 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-19T03:04:52.686 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-19T03:04:53.774 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-19T03:04:53.800 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-700.gb4843b 12/12 2024-03-19T03:04:56.304 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-700.gb4843b 12/12 2024-03-19T03:04:56.304 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-700.gb4843b 1/12 2024-03-19T03:04:56.304 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-19T03:04:56.304 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-19T03:04:56.304 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-19T03:04:56.304 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-19T03:04:56.304 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-19T03:04:56.305 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-19T03:04:56.305 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-19T03:04:56.305 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-19T03:04:56.305 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-19T03:04:56.305 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-19T03:04:56.712 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-19T03:04:56.712 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:04:56.712 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-19T03:04:56.712 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local-2:17.2.7-700.gb4843bd7.el9.noarch 2024-03-19T03:04:56.712 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-19T03:04:56.712 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-19T03:04:56.712 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-19T03:04:56.712 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-19T03:04:56.712 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-19T03:04:56.712 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-19T03:04:56.713 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-19T03:04:56.713 INFO:teuthology.orchestra.run.smithi059.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-19T03:04:56.713 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-19T03:04:56.713 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-19T03:04:56.713 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-19T03:04:56.713 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:04:56.713 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T03:04:57.043 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-rook 2024-03-19T03:04:57.620 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:10:03 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T03:04:57.777 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T03:04:57.778 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================= 2024-03-19T03:04:57.778 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-03-19T03:04:57.778 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================= 2024-03-19T03:04:57.778 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-19T03:04:57.778 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook noarch 2:17.2.7-700.gb4843bd7.el9 ceph-noarch 47 k 2024-03-19T03:04:57.779 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-03-19T03:04:57.779 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-19T03:04:57.779 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-19T03:04:57.779 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-19T03:04:57.779 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-19T03:04:57.779 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-19T03:04:57.779 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-19T03:04:57.779 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-19T03:04:57.779 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-19T03:04:57.779 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-19T03:04:57.779 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-19T03:04:57.779 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-19T03:04:57.779 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-19T03:04:57.780 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:04:57.780 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-19T03:04:57.780 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================= 2024-03-19T03:04:57.780 INFO:teuthology.orchestra.run.smithi059.stdout:Install 13 Packages 2024-03-19T03:04:57.780 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:04:57.780 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 2.0 M 2024-03-19T03:04:57.780 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 26 M 2024-03-19T03:04:57.781 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-19T03:04:58.274 INFO:teuthology.orchestra.run.smithi059.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 197 kB/s | 24 kB 00:00 2024-03-19T03:04:58.299 INFO:teuthology.orchestra.run.smithi059.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 121 kB/s | 18 kB 00:00 2024-03-19T03:04:58.341 INFO:teuthology.orchestra.run.smithi059.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-19T03:04:58.366 INFO:teuthology.orchestra.run.smithi059.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-19T03:04:58.392 INFO:teuthology.orchestra.run.smithi059.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-19T03:04:58.417 INFO:teuthology.orchestra.run.smithi059.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-19T03:04:58.442 INFO:teuthology.orchestra.run.smithi059.stdout:(7/13): ceph-mgr-rook-17.2.7-700.gb4843bd7.el9. 163 kB/s | 47 kB 00:00 2024-03-19T03:04:58.742 INFO:teuthology.orchestra.run.smithi059.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 46 kB/s | 15 kB 00:00 2024-03-19T03:04:58.884 INFO:teuthology.orchestra.run.smithi059.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 65 kB/s | 32 kB 00:00 2024-03-19T03:04:59.835 INFO:teuthology.orchestra.run.smithi059.stdout:(10/13): python3-google-auth-2.28.0-1.el9.noarc 146 kB/s | 203 kB 00:01 2024-03-19T03:05:02.476 INFO:teuthology.orchestra.run.smithi059.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 16 kB/s | 59 kB 00:03 2024-03-19T03:05:02.810 INFO:teuthology.orchestra.run.smithi059.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 257 kB/s | 1.0 MB 00:04 2024-03-19T03:05:02.927 INFO:teuthology.orchestra.run.smithi059.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 29 kB/s | 90 kB 00:03 2024-03-19T03:05:02.931 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-19T03:05:02.932 INFO:teuthology.orchestra.run.smithi059.stdout:Total 406 kB/s | 2.0 MB 00:05 2024-03-19T03:05:02.932 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-19T03:05:02.948 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-19T03:05:02.948 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-19T03:05:03.049 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-19T03:05:03.049 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-19T03:05:03.605 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-19T03:05:03.788 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-19T03:05:03.846 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-19T03:05:03.910 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-19T03:05:03.946 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-19T03:05:03.988 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-19T03:05:04.086 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-19T03:05:04.194 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-19T03:05:04.248 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-19T03:05:05.007 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-19T03:05:05.102 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-19T03:05:05.135 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-19T03:05:05.182 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-19T03:05:05.206 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-rook-2:17.2.7-700.gb4843bd7.el9.noarch 13/13 2024-03-19T03:05:05.899 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-700.gb4843bd7.el9.noarch 13/13 2024-03-19T03:05:05.899 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-rook-2:17.2.7-700.gb4843bd7.el9.noarch 1/13 2024-03-19T03:05:05.899 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-19T03:05:05.899 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-19T03:05:05.899 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-19T03:05:05.899 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-19T03:05:05.899 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-19T03:05:05.900 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-19T03:05:05.900 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-19T03:05:05.900 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-19T03:05:05.900 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-19T03:05:05.900 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-19T03:05:05.900 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-19T03:05:06.252 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-19T03:05:06.252 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:06.252 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-19T03:05:06.253 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook-2:17.2.7-700.gb4843bd7.el9.noarch 2024-03-19T03:05:06.253 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-19T03:05:06.253 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-19T03:05:06.253 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-19T03:05:06.253 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-19T03:05:06.253 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-19T03:05:06.253 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-19T03:05:06.253 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-19T03:05:06.253 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-19T03:05:06.253 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-19T03:05:06.253 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-19T03:05:06.253 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-19T03:05:06.253 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-19T03:05:06.254 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:06.254 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T03:05:06.476 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-cephadm 2024-03-19T03:05:07.050 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:10:13 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T03:05:07.206 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T03:05:07.207 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================= 2024-03-19T03:05:07.207 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-03-19T03:05:07.207 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================= 2024-03-19T03:05:07.207 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-19T03:05:07.207 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm noarch 2:17.2.7-700.gb4843bd7.el9 ceph-noarch 123 k 2024-03-19T03:05:07.208 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-03-19T03:05:07.208 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-19T03:05:07.208 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-19T03:05:07.208 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-19T03:05:07.208 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-19T03:05:07.208 INFO:teuthology.orchestra.run.smithi059.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-19T03:05:07.208 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:07.208 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-19T03:05:07.208 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================= 2024-03-19T03:05:07.208 INFO:teuthology.orchestra.run.smithi059.stdout:Install 6 Packages 2024-03-19T03:05:07.208 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:07.209 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 6.8 M 2024-03-19T03:05:07.209 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 33 M 2024-03-19T03:05:07.209 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-19T03:05:07.769 INFO:teuthology.orchestra.run.smithi059.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-19T03:05:07.844 INFO:teuthology.orchestra.run.smithi059.stdout:(2/6): ceph-mgr-cephadm-17.2.7-700.gb4843bd7.el 342 kB/s | 123 kB 00:00 2024-03-19T03:05:08.003 INFO:teuthology.orchestra.run.smithi059.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 477 kB/s | 248 kB 00:00 2024-03-19T03:05:08.195 INFO:teuthology.orchestra.run.smithi059.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 165 kB/s | 58 kB 00:00 2024-03-19T03:05:08.311 INFO:teuthology.orchestra.run.smithi059.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 165 kB/s | 51 kB 00:00 2024-03-19T03:05:08.370 INFO:teuthology.orchestra.run.smithi059.stdout:(6/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 912 kB/s | 548 kB 00:00 2024-03-19T03:05:08.371 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-19T03:05:08.371 INFO:teuthology.orchestra.run.smithi059.stdout:Total 5.8 MB/s | 6.8 MB 00:01 2024-03-19T03:05:08.386 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-19T03:05:08.398 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-19T03:05:08.398 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-19T03:05:08.493 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-19T03:05:08.493 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-19T03:05:08.801 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-19T03:05:08.974 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-19T03:05:09.022 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-19T03:05:09.581 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-19T03:05:09.668 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-19T03:05:09.730 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-19T03:05:09.758 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-700.gb4843bd7.el9.noarch 6/6 2024-03-19T03:05:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-700.gb4843bd7.el9.noarch 6/6 2024-03-19T03:05:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-700.gb4843bd7.el9.noarch 1/6 2024-03-19T03:05:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-19T03:05:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-19T03:05:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-19T03:05:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-19T03:05:10.825 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-19T03:05:10.825 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:10.825 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-19T03:05:10.825 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm-2:17.2.7-700.gb4843bd7.el9.noarch 2024-03-19T03:05:10.825 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-19T03:05:10.825 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-19T03:05:10.825 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-19T03:05:10.825 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-19T03:05:10.825 INFO:teuthology.orchestra.run.smithi059.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-19T03:05:10.825 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:10.825 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T03:05:11.052 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-fuse 2024-03-19T03:05:11.623 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:10:17 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T03:05:11.778 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T03:05:11.779 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:05:11.779 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-03-19T03:05:11.779 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:05:11.779 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-19T03:05:11.779 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 812 k 2024-03-19T03:05:11.779 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-03-19T03:05:11.779 INFO:teuthology.orchestra.run.smithi059.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-19T03:05:11.779 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:11.780 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-19T03:05:11.780 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:05:11.780 INFO:teuthology.orchestra.run.smithi059.stdout:Install 2 Packages 2024-03-19T03:05:11.780 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:11.780 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 890 k 2024-03-19T03:05:11.780 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 2.6 M 2024-03-19T03:05:11.781 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-19T03:05:11.905 INFO:teuthology.orchestra.run.smithi059.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 635 kB/s | 79 kB 00:00 2024-03-19T03:05:12.080 INFO:teuthology.orchestra.run.smithi059.stdout:(2/2): ceph-fuse-17.2.7-700.gb4843bd7.el9.x86_6 2.7 MB/s | 812 kB 00:00 2024-03-19T03:05:12.081 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-19T03:05:12.081 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.9 MB/s | 890 kB 00:00 2024-03-19T03:05:12.081 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-19T03:05:12.103 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-19T03:05:12.103 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-19T03:05:12.177 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-19T03:05:12.177 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-19T03:05:12.363 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-19T03:05:12.524 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-19T03:05:12.598 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-fuse-2:17.2.7-700.gb4843bd7.el9.x86_64 2/2 2024-03-19T03:05:13.276 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-fuse-2:17.2.7-700.gb4843bd7.el9.x86_64 2/2 2024-03-19T03:05:13.276 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-fuse-2:17.2.7-700.gb4843bd7.el9.x86_64 1/2 2024-03-19T03:05:13.590 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-19T03:05:13.590 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:13.590 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-19T03:05:13.590 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse-2:17.2.7-700.gb4843bd7.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-19T03:05:13.590 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:13.591 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T03:05:13.815 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librados-devel 2024-03-19T03:05:14.393 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:10:20 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T03:05:14.548 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T03:05:14.549 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:05:14.549 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-03-19T03:05:14.549 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:05:14.549 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-19T03:05:14.549 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 126 k 2024-03-19T03:05:14.549 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:14.549 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-19T03:05:14.549 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:05:14.549 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-03-19T03:05:14.550 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:14.550 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 126 k 2024-03-19T03:05:14.550 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 448 k 2024-03-19T03:05:14.550 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-19T03:05:14.775 INFO:teuthology.orchestra.run.smithi059.stdout:librados-devel-17.2.7-700.gb4843bd7.el9.x86_64. 560 kB/s | 126 kB 00:00 2024-03-19T03:05:14.775 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-19T03:05:14.776 INFO:teuthology.orchestra.run.smithi059.stdout:Total 556 kB/s | 126 kB 00:00 2024-03-19T03:05:14.776 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-19T03:05:14.782 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-19T03:05:14.782 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-19T03:05:14.840 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-19T03:05:14.840 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-19T03:05:15.120 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-19T03:05:15.218 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librados-devel-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T03:05:15.671 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados-devel-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T03:05:15.957 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados-devel-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T03:05:15.958 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:15.958 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-19T03:05:15.958 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T03:05:15.958 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:15.958 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T03:05:16.194 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs2 2024-03-19T03:05:16.762 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:10:22 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T03:05:16.861 INFO:teuthology.orchestra.run.smithi059.stdout:Package libcephfs2-2:17.2.7-700.gb4843bd7.el9.x86_64 is already installed. 2024-03-19T03:05:16.916 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T03:05:16.917 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-03-19T03:05:16.918 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T03:05:16.974 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs-devel 2024-03-19T03:05:17.547 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:10:23 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T03:05:17.702 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T03:05:17.703 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:05:17.703 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-03-19T03:05:17.703 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:05:17.703 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-19T03:05:17.703 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 25 k 2024-03-19T03:05:17.703 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:17.703 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-19T03:05:17.703 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:05:17.703 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-03-19T03:05:17.704 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:17.704 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 25 k 2024-03-19T03:05:17.704 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 110 k 2024-03-19T03:05:17.704 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-19T03:05:17.870 INFO:teuthology.orchestra.run.smithi059.stdout:libcephfs-devel-17.2.7-700.gb4843bd7.el9.x86_64 150 kB/s | 25 kB 00:00 2024-03-19T03:05:17.870 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-19T03:05:17.870 INFO:teuthology.orchestra.run.smithi059.stdout:Total 149 kB/s | 25 kB 00:00 2024-03-19T03:05:17.871 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-19T03:05:17.874 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-19T03:05:17.874 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-19T03:05:17.875 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-19T03:05:17.876 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-19T03:05:17.954 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-19T03:05:18.070 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs-devel-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T03:05:18.455 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T03:05:18.715 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs-devel-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T03:05:18.715 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:18.715 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-19T03:05:18.715 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T03:05:18.715 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:18.715 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T03:05:18.874 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librados2 2024-03-19T03:05:19.447 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:10:25 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T03:05:19.548 INFO:teuthology.orchestra.run.smithi059.stdout:Package librados2-2:17.2.7-700.gb4843bd7.el9.x86_64 is already installed. 2024-03-19T03:05:19.602 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T03:05:19.603 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-03-19T03:05:19.603 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T03:05:19.657 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librbd1 2024-03-19T03:05:20.226 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:10:26 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T03:05:20.325 INFO:teuthology.orchestra.run.smithi059.stdout:Package librbd1-2:17.2.7-700.gb4843bd7.el9.x86_64 is already installed. 2024-03-19T03:05:20.381 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T03:05:20.382 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-03-19T03:05:20.382 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T03:05:20.441 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rados 2024-03-19T03:05:21.009 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:10:27 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T03:05:21.110 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rados-2:17.2.7-700.gb4843bd7.el9.x86_64 is already installed. 2024-03-19T03:05:21.164 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T03:05:21.165 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-03-19T03:05:21.166 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T03:05:21.225 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rgw 2024-03-19T03:05:21.794 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:10:27 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T03:05:21.895 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rgw-2:17.2.7-700.gb4843bd7.el9.x86_64 is already installed. 2024-03-19T03:05:21.950 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T03:05:21.952 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-03-19T03:05:21.952 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T03:05:22.005 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-cephfs 2024-03-19T03:05:22.572 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:10:28 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T03:05:22.673 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-cephfs-2:17.2.7-700.gb4843bd7.el9.x86_64 is already installed. 2024-03-19T03:05:22.727 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T03:05:22.728 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-03-19T03:05:22.728 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T03:05:22.786 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rbd 2024-03-19T03:05:23.356 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:10:29 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T03:05:23.456 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rbd-2:17.2.7-700.gb4843bd7.el9.x86_64 is already installed. 2024-03-19T03:05:23.511 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T03:05:23.513 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-03-19T03:05:23.513 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T03:05:23.569 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-fuse 2024-03-19T03:05:24.138 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:10:30 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T03:05:24.295 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T03:05:24.296 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:05:24.296 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-03-19T03:05:24.296 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:05:24.296 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-19T03:05:24.296 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 82 k 2024-03-19T03:05:24.296 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:24.296 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-19T03:05:24.297 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:05:24.297 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-03-19T03:05:24.297 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:24.297 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 82 k 2024-03-19T03:05:24.297 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 222 k 2024-03-19T03:05:24.297 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-19T03:05:24.608 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-fuse-17.2.7-700.gb4843bd7.el9.x86_64.rpm 261 kB/s | 82 kB 00:00 2024-03-19T03:05:24.609 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-19T03:05:24.609 INFO:teuthology.orchestra.run.smithi059.stdout:Total 260 kB/s | 82 kB 00:00 2024-03-19T03:05:24.609 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-19T03:05:24.617 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-19T03:05:24.617 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-19T03:05:24.679 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-19T03:05:24.679 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-19T03:05:24.833 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-19T03:05:24.942 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-fuse-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T03:05:25.327 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-fuse-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T03:05:25.602 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-fuse-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T03:05:25.602 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:25.602 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-19T03:05:25.602 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T03:05:25.602 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:25.602 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T03:05:25.796 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-mirror 2024-03-19T03:05:26.378 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:10:32 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T03:05:26.539 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T03:05:26.540 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:05:26.540 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-03-19T03:05:26.540 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:05:26.540 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-19T03:05:26.540 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 2.9 M 2024-03-19T03:05:26.540 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:26.540 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-19T03:05:26.541 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:05:26.541 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-03-19T03:05:26.541 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:26.541 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 2.9 M 2024-03-19T03:05:26.541 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 12 M 2024-03-19T03:05:26.541 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-19T03:05:26.945 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-mirror-17.2.7-700.gb4843bd7.el9.x86_64.rpm 7.3 MB/s | 2.9 MB 00:00 2024-03-19T03:05:26.945 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-19T03:05:26.946 INFO:teuthology.orchestra.run.smithi059.stdout:Total 7.2 MB/s | 2.9 MB 00:00 2024-03-19T03:05:26.946 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-19T03:05:26.956 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-19T03:05:26.956 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-19T03:05:27.030 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-19T03:05:27.030 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-19T03:05:27.464 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-19T03:05:27.529 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-mirror-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T03:05:27.553 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-mirror-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T03:05:27.553 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:05:27.554 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-19T03:05:27.554 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T03:05:27.554 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T03:05:27.554 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:28.697 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-mirror-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T03:05:28.697 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:28.697 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-19T03:05:28.697 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T03:05:28.697 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:28.697 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T03:05:28.927 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-nbd 2024-03-19T03:05:29.503 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:10:35 ago on Tue 19 Mar 2024 02:54:54 AM UTC. 2024-03-19T03:05:29.661 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-19T03:05:29.662 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:05:29.662 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-03-19T03:05:29.662 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:05:29.662 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-19T03:05:29.662 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd x86_64 2:17.2.7-700.gb4843bd7.el9 ceph 165 k 2024-03-19T03:05:29.662 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:29.662 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-19T03:05:29.662 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-19T03:05:29.662 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-03-19T03:05:29.662 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:29.662 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 165 k 2024-03-19T03:05:29.663 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 477 k 2024-03-19T03:05:29.663 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-19T03:05:29.889 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-nbd-17.2.7-700.gb4843bd7.el9.x86_64.rpm 730 kB/s | 165 kB 00:00 2024-03-19T03:05:29.890 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-19T03:05:29.890 INFO:teuthology.orchestra.run.smithi059.stdout:Total 724 kB/s | 165 kB 00:00 2024-03-19T03:05:29.890 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-19T03:05:29.898 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-19T03:05:29.898 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-19T03:05:29.956 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-19T03:05:29.956 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-19T03:05:30.114 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-19T03:05:30.215 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-nbd-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T03:05:30.683 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-nbd-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T03:05:31.005 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-nbd-2:17.2.7-700.gb4843bd7.el9.x86_64 1/1 2024-03-19T03:05:31.005 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:31.005 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-19T03:05:31.005 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd-2:17.2.7-700.gb4843bd7.el9.x86_64 2024-03-19T03:05:31.005 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:05:31.005 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-19T03:05:31.217 DEBUG:teuthology.parallel:result is None 2024-03-19T03:05:31.218 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-19T03:05:31.218 INFO:teuthology.packaging:ref: None 2024-03-19T03:05:31.218 INFO:teuthology.packaging:tag: None 2024-03-19T03:05:31.218 INFO:teuthology.packaging:branch: quincy 2024-03-19T03:05:31.218 INFO:teuthology.packaging:sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T03:05:31.219 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-03-19T03:05:31.476 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-19T03:05:31.497 INFO:teuthology.orchestra.run.smithi059.stdout:17.2.7-700.gb4843bd7.el9 2024-03-19T03:05:31.497 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-700.gb4843bd7.el9 2024-03-19T03:05:31.498 INFO:teuthology.task.install:The correct ceph version 17.2.7-700.gb4843bd7 is installed. 2024-03-19T03:05:31.499 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-19T03:05:31.500 INFO:teuthology.packaging:ref: None 2024-03-19T03:05:31.500 INFO:teuthology.packaging:tag: None 2024-03-19T03:05:31.500 INFO:teuthology.packaging:branch: quincy 2024-03-19T03:05:31.500 INFO:teuthology.packaging:sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T03:05:31.500 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-03-19T03:05:31.737 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-19T03:05:31.763 INFO:teuthology.orchestra.run.smithi176.stdout:17.2.7-700.gb4843bd7.el9 2024-03-19T03:05:31.764 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-700.gb4843bd7.el9 2024-03-19T03:05:31.764 INFO:teuthology.task.install:The correct ceph version 17.2.7-700.gb4843bd7 is installed. 2024-03-19T03:05:31.765 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-19T03:05:31.766 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-19T03:05:31.766 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-19T03:05:31.800 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-19T03:05:31.800 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-19T03:05:31.837 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-19T03:05:31.837 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-19T03:05:31.837 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/daemon-helper 2024-03-19T03:05:31.867 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-19T03:05:31.935 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-19T03:05:31.935 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/daemon-helper 2024-03-19T03:05:31.967 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-19T03:05:32.039 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-19T03:05:32.039 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-19T03:05:32.039 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-19T03:05:32.067 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-19T03:05:32.134 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-19T03:05:32.134 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-19T03:05:32.165 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-19T03:05:32.234 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-19T03:05:32.235 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-19T03:05:32.235 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/stdin-killer 2024-03-19T03:05:32.262 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-19T03:05:32.329 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-19T03:05:32.329 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/stdin-killer 2024-03-19T03:05:32.359 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-19T03:05:32.429 INFO:teuthology.run_tasks:Running task print... 2024-03-19T03:05:32.441 INFO:teuthology.task.print:**** done install task... 2024-03-19T03:05:32.442 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-19T03:05:32.559 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'PG_AVAILABILITY', 'PG_DEGRADED', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'} 2024-03-19T03:05:32.560 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-03-19T03:05:32.560 INFO:tasks.cephadm:Cluster fsid is 8c8ae530-e59d-11ee-95c9-87774f69a715 2024-03-19T03:05:32.560 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-19T03:05:32.560 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-03-19T03:05:32.560 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi059': '172.21.15.59', 'mon.smithi176': '172.21.15.176'} 2024-03-19T03:05:32.560 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-19T03:05:32.561 DEBUG:teuthology.orchestra.run.smithi059:> sudo hostname $(hostname -s) 2024-03-19T03:05:32.588 DEBUG:teuthology.orchestra.run.smithi176:> sudo hostname $(hostname -s) 2024-03-19T03:05:32.619 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-03-19T03:05:32.619 DEBUG:teuthology.orchestra.run.smithi059:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-19T03:05:32.856 INFO:teuthology.orchestra.run.smithi059.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Mar 19 03:05 /home/ubuntu/cephtest/cephadm 2024-03-19T03:05:32.856 DEBUG:teuthology.orchestra.run.smithi176:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-19T03:05:33.147 INFO:teuthology.orchestra.run.smithi176.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Mar 19 03:05 /home/ubuntu/cephtest/cephadm 2024-03-19T03:05:33.148 DEBUG:teuthology.orchestra.run.smithi059:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-19T03:05:33.165 DEBUG:teuthology.orchestra.run.smithi176:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-19T03:05:33.210 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-03-19T03:05:33.211 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-03-19T03:05:33.213 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-03-19T03:05:34.024 INFO:teuthology.orchestra.run.smithi176.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-19T03:05:34.132 INFO:teuthology.orchestra.run.smithi059.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-19T03:06:03.242 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-19T03:06:03.243 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_version": "ceph version 17.2.7-700-gb4843bd7 (b4843bd759866f9ee1bdfb1dfa084ec7d085b910) quincy (stable)", 2024-03-19T03:06:03.243 INFO:teuthology.orchestra.run.smithi176.stdout: "image_id": "810955603526f1a5056294fef84995d43fccb8e7018e3b8031db5047002886ab", 2024-03-19T03:06:03.243 INFO:teuthology.orchestra.run.smithi176.stdout: "repo_digests": [ 2024-03-19T03:06:03.243 INFO:teuthology.orchestra.run.smithi176.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881" 2024-03-19T03:06:03.243 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-19T03:06:03.243 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-19T03:06:06.633 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-19T03:06:06.633 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_version": "ceph version 17.2.7-700-gb4843bd7 (b4843bd759866f9ee1bdfb1dfa084ec7d085b910) quincy (stable)", 2024-03-19T03:06:06.633 INFO:teuthology.orchestra.run.smithi059.stdout: "image_id": "810955603526f1a5056294fef84995d43fccb8e7018e3b8031db5047002886ab", 2024-03-19T03:06:06.633 INFO:teuthology.orchestra.run.smithi059.stdout: "repo_digests": [ 2024-03-19T03:06:06.633 INFO:teuthology.orchestra.run.smithi059.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881" 2024-03-19T03:06:06.633 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-03-19T03:06:06.633 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-19T03:06:06.655 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph 2024-03-19T03:06:06.706 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph 2024-03-19T03:06:06.741 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 777 /etc/ceph 2024-03-19T03:06:06.768 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 777 /etc/ceph 2024-03-19T03:06:06.811 INFO:tasks.cephadm:Writing seed config... 2024-03-19T03:06:06.812 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-03-19T03:06:06.812 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-03-19T03:06:06.812 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-03-19T03:06:06.812 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-03-19T03:06:06.812 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-03-19T03:06:06.812 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-19T03:06:06.812 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-19T03:06:06.813 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-03-19T03:06:06.813 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-03-19T03:06:06.813 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-19T03:06:06.813 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-19T03:06:06.813 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-03-19T03:06:06.813 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-19T03:06:06.813 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-19T03:06:06.813 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-19T03:06:06.813 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-19T03:06:06.813 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-19T03:06:06.813 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-19T03:06:06.813 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-19T03:06:06.813 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-19T03:06:06.813 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-19T03:06:06.814 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-19T03:06:06.814 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-19T03:06:06.814 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-03-19T03:06:06.814 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-19T03:06:06.814 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-19T03:06:06.814 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-19T03:06:06.814 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-19T03:06:06.814 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-19T03:06:06.814 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-19T03:06:06.814 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-19T03:06:06.814 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-19T03:06:06.814 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-19T03:06:06.815 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-19T03:06:06.815 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-19T03:06:06.815 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-19T03:06:06.815 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-19T03:06:06.815 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-19T03:06:06.815 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-19T03:06:06.815 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-19T03:06:06.816 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-19T03:06:06.816 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-19T03:06:06.830 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 8c8ae530-e59d-11ee-95c9-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-19T03:06:06.831 DEBUG:teuthology.orchestra.run.smithi059:mon.smithi059> sudo journalctl -f -n 0 -u ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@mon.smithi059.service 2024-03-19T03:06:06.872 INFO:tasks.cephadm:Bootstrapping... 2024-03-19T03:06:06.873 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.59 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-19T03:06:07.080 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-19T03:06:07.081 INFO:teuthology.orchestra.run.smithi059.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '8c8ae530-e59d-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.59', '--skip-admin-label'] 2024-03-19T03:06:07.107 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-19T03:06:07.107 INFO:teuthology.orchestra.run.smithi059.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-19T03:06:07.108 INFO:teuthology.orchestra.run.smithi059.stdout:Verifying podman|docker is present... 2024-03-19T03:06:07.130 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-19T03:06:07.130 INFO:teuthology.orchestra.run.smithi059.stdout:Verifying lvm2 is present... 2024-03-19T03:06:07.130 INFO:teuthology.orchestra.run.smithi059.stdout:Verifying time synchronization is in place... 2024-03-19T03:06:07.138 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-19T03:06:07.138 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-19T03:06:07.144 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-19T03:06:07.145 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout inactive 2024-03-19T03:06:07.151 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout enabled 2024-03-19T03:06:07.157 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout active 2024-03-19T03:06:07.157 INFO:teuthology.orchestra.run.smithi059.stdout:Unit chronyd.service is enabled and running 2024-03-19T03:06:07.157 INFO:teuthology.orchestra.run.smithi059.stdout:Repeating the final host check... 2024-03-19T03:06:07.179 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-19T03:06:07.179 INFO:teuthology.orchestra.run.smithi059.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-19T03:06:07.179 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl is present 2024-03-19T03:06:07.179 INFO:teuthology.orchestra.run.smithi059.stdout:lvcreate is present 2024-03-19T03:06:07.186 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-19T03:06:07.186 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-19T03:06:07.192 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-19T03:06:07.192 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout inactive 2024-03-19T03:06:07.198 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout enabled 2024-03-19T03:06:07.205 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout active 2024-03-19T03:06:07.205 INFO:teuthology.orchestra.run.smithi059.stdout:Unit chronyd.service is enabled and running 2024-03-19T03:06:07.205 INFO:teuthology.orchestra.run.smithi059.stdout:Host looks OK 2024-03-19T03:06:07.205 INFO:teuthology.orchestra.run.smithi059.stdout:Cluster fsid: 8c8ae530-e59d-11ee-95c9-87774f69a715 2024-03-19T03:06:07.205 INFO:teuthology.orchestra.run.smithi059.stdout:Acquiring lock 140023745061984 on /run/cephadm/8c8ae530-e59d-11ee-95c9-87774f69a715.lock 2024-03-19T03:06:07.205 INFO:teuthology.orchestra.run.smithi059.stdout:Lock 140023745061984 acquired on /run/cephadm/8c8ae530-e59d-11ee-95c9-87774f69a715.lock 2024-03-19T03:06:07.206 INFO:teuthology.orchestra.run.smithi059.stdout:Verifying IP 172.21.15.59 port 3300 ... 2024-03-19T03:06:07.206 INFO:teuthology.orchestra.run.smithi059.stdout:Verifying IP 172.21.15.59 port 6789 ... 2024-03-19T03:06:07.206 INFO:teuthology.orchestra.run.smithi059.stdout:Base mon IP(s) is [172.21.15.59:3300, 172.21.15.59:6789], mon addrv is [v2:172.21.15.59:3300,v1:172.21.15.59:6789] 2024-03-19T03:06:07.210 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.59 metric 100 2024-03-19T03:06:07.210 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.59 metric 100 2024-03-19T03:06:07.212 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-19T03:06:07.212 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-19T03:06:07.212 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-19T03:06:07.215 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-19T03:06:07.215 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-19T03:06:07.215 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-19T03:06:07.215 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-19T03:06:07.215 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-03-19T03:06:07.215 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-19T03:06:07.216 INFO:teuthology.orchestra.run.smithi059.stdout:Mon IP `172.21.15.59` is in CIDR network `172.21.0.0/20` 2024-03-19T03:06:07.216 INFO:teuthology.orchestra.run.smithi059.stdout:Mon IP `172.21.15.59` is in CIDR network `172.21.0.0/20` 2024-03-19T03:06:07.216 INFO:teuthology.orchestra.run.smithi059.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-19T03:06:07.216 INFO:teuthology.orchestra.run.smithi059.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-19T03:06:07.217 INFO:teuthology.orchestra.run.smithi059.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-19T03:06:07.696 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stdout 810955603526f1a5056294fef84995d43fccb8e7018e3b8031db5047002886ab 2024-03-19T03:06:07.697 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-19T03:06:07.697 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stderr Getting image source signatures 2024-03-19T03:06:07.697 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stderr Copying blob sha256:8fd90764ba6470340246d8c1acbaea8f5897daa2c687916ed1aaf8e998a13cfe 2024-03-19T03:06:07.697 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-19T03:06:07.697 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stderr Copying config sha256:810955603526f1a5056294fef84995d43fccb8e7018e3b8031db5047002886ab 2024-03-19T03:06:07.697 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-19T03:06:08.563 INFO:teuthology.orchestra.run.smithi059.stdout:ceph: stdout ceph version 17.2.7-700-gb4843bd7 (b4843bd759866f9ee1bdfb1dfa084ec7d085b910) quincy (stable) 2024-03-19T03:06:08.563 INFO:teuthology.orchestra.run.smithi059.stdout:Ceph version: ceph version 17.2.7-700-gb4843bd7 (b4843bd759866f9ee1bdfb1dfa084ec7d085b910) quincy (stable) 2024-03-19T03:06:08.563 INFO:teuthology.orchestra.run.smithi059.stdout:Extracting ceph user uid/gid from container image... 2024-03-19T03:06:09.368 INFO:teuthology.orchestra.run.smithi059.stdout:stat: stdout 167 167 2024-03-19T03:06:09.368 INFO:teuthology.orchestra.run.smithi059.stdout:Creating initial keys... 2024-03-19T03:06:10.198 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph-authtool: stdout AQAhAfllNPYoKBAAkEopzjku8b12LWIOSbPx6A== 2024-03-19T03:06:11.070 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph-authtool: stdout AQAiAfllhrIBHRAAWfCzYEib9JPK4YFz8BI0ng== 2024-03-19T03:06:12.001 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph-authtool: stdout AQAjAfllBIj+GRAA+lR3SgSDUYJIvHLDGxvGww== 2024-03-19T03:06:12.002 INFO:teuthology.orchestra.run.smithi059.stdout:Creating initial monmap... 2024-03-19T03:06:12.848 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-19T03:06:12.848 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-03-19T03:06:12.848 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 8c8ae530-e59d-11ee-95c9-87774f69a715 2024-03-19T03:06:12.848 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-19T03:06:12.849 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool for smithi059 [v2:172.21.15.59:3300,v1:172.21.15.59:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-19T03:06:12.849 INFO:teuthology.orchestra.run.smithi059.stdout:setting min_mon_release = octopus 2024-03-19T03:06:12.849 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/monmaptool: set fsid to 8c8ae530-e59d-11ee-95c9-87774f69a715 2024-03-19T03:06:12.849 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-19T03:06:12.849 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:06:12.849 INFO:teuthology.orchestra.run.smithi059.stdout:Creating mon... 2024-03-19T03:06:13.762 INFO:teuthology.orchestra.run.smithi059.stdout:create mon.smithi059 on 2024-03-19T03:06:13.967 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-19T03:06:14.110 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-19T03:06:14.256 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-8c8ae530-e59d-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-8c8ae530-e59d-11ee-95c9-87774f69a715.target. 2024-03-19T03:06:14.256 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-8c8ae530-e59d-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-8c8ae530-e59d-11ee-95c9-87774f69a715.target. 2024-03-19T03:06:14.424 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@mon.smithi059 2024-03-19T03:06:14.424 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@mon.smithi059.service: Unit ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@mon.smithi059.service not loaded. 2024-03-19T03:06:14.580 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8c8ae530-e59d-11ee-95c9-87774f69a715.target.wants/ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@mon.smithi059.service → /etc/systemd/system/ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@.service. 2024-03-19T03:06:15.067 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:15 smithi059 podman[28731]: 2024-03-19 03:06:15.01504345 +0000 UTC m=+0.313776647 container start 9449ca80148e10c531825073881fe12d1ab9836ea593daa4b2f11bc74100075b (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059, RELEASE=quincy-b4843bd, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-19T03:06:15.067 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:15 smithi059 ceph-mon[28750]: mkfs 8c8ae530-e59d-11ee-95c9-87774f69a715 2024-03-19T03:06:15.126 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-19T03:06:15.126 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout disabled 2024-03-19T03:06:15.134 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-19T03:06:15.134 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout inactive 2024-03-19T03:06:15.134 INFO:teuthology.orchestra.run.smithi059.stdout:firewalld.service is not enabled 2024-03-19T03:06:15.134 INFO:teuthology.orchestra.run.smithi059.stdout:Not possible to enable service . firewalld.service is not available 2024-03-19T03:06:15.135 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for mon to start... 2024-03-19T03:06:15.135 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for mon... 2024-03-19T03:06:15.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:15 smithi059 ceph-mon[28750]: mon.smithi059 is new leader, mons smithi059 in quorum (ranks 0) 2024-03-19T03:06:15.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:15 smithi059 bash[28731]: 9449ca80148e10c531825073881fe12d1ab9836ea593daa4b2f11bc74100075b 2024-03-19T03:06:15.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:15 smithi059 systemd[1]: Started Ceph mon.smithi059 for 8c8ae530-e59d-11ee-95c9-87774f69a715. 2024-03-19T03:06:16.321 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:16 smithi059 ceph-mon[28750]: mon.smithi059 is new leader, mons smithi059 in quorum (ranks 0) 2024-03-19T03:06:16.321 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:16 smithi059 ceph-mon[28750]: monmap e1: 1 mons at {smithi059=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-19T03:06:16.321 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:16 smithi059 ceph-mon[28750]: fsmap 2024-03-19T03:06:16.321 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:16 smithi059 ceph-mon[28750]: osdmap e1: 0 total, 0 up, 0 in 2024-03-19T03:06:16.322 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:16 smithi059 ceph-mon[28750]: mgrmap e1: no daemons active 2024-03-19T03:06:16.322 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:16 smithi059 ceph-mon[28750]: from='client.? 172.21.15.59:0/1899745089' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-19T03:06:16.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout cluster: 2024-03-19T03:06:16.332 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout id: 8c8ae530-e59d-11ee-95c9-87774f69a715 2024-03-19T03:06:16.332 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-19T03:06:16.332 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-03-19T03:06:16.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout services: 2024-03-19T03:06:16.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi059 (age 0.63358s) 2024-03-19T03:06:16.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-19T03:06:16.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-19T03:06:16.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-03-19T03:06:16.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout data: 2024-03-19T03:06:16.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-19T03:06:16.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-19T03:06:16.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-19T03:06:16.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout pgs: 2024-03-19T03:06:16.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-03-19T03:06:16.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.617+0000 7fbac0bae700 1 Processor -- start 2024-03-19T03:06:16.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.617+0000 7fbac0bae700 1 -- start start 2024-03-19T03:06:16.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.617+0000 7fbac0bae700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbabc078d20 0x7fbabc077220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:16.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.617+0000 7fbac0bae700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc079130 con 0x7fbabc078d20 2024-03-19T03:06:16.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.618+0000 7fbaba59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbabc078d20 0x7fbabc077220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:16.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.618+0000 7fbaba59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbabc078d20 0x7fbabc077220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35884/0 (socket says 172.21.15.59:35884) 2024-03-19T03:06:16.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.618+0000 7fbaba59c700 1 -- 172.21.15.59:0/87010011 learned_addr learned my addr 172.21.15.59:0/87010011 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:16.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.618+0000 7fbaba59c700 1 -- 172.21.15.59:0/87010011 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbabc077760 con 0x7fbabc078d20 2024-03-19T03:06:16.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.618+0000 7fbaba59c700 1 --2- 172.21.15.59:0/87010011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbabc078d20 0x7fbabc077220 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbaac0098e0 tx=0x7fbaac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef419491b80af9f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:16.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.618+0000 7fbab959a700 1 -- 172.21.15.59:0/87010011 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbaac00b860 con 0x7fbabc078d20 2024-03-19T03:06:16.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.618+0000 7fbab959a700 1 -- 172.21.15.59:0/87010011 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaac00b9c0 con 0x7fbabc078d20 2024-03-19T03:06:16.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.618+0000 7fbab959a700 1 -- 172.21.15.59:0/87010011 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbaac00bc90 con 0x7fbabc078d20 2024-03-19T03:06:16.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.619+0000 7fbac0bae700 1 -- 172.21.15.59:0/87010011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbabc078d20 msgr2=0x7fbabc077220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:16.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.619+0000 7fbac0bae700 1 --2- 172.21.15.59:0/87010011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbabc078d20 0x7fbabc077220 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbaac0098e0 tx=0x7fbaac009250 comp rx=0 tx=0).stop 2024-03-19T03:06:16.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.619+0000 7fbac0bae700 1 -- 172.21.15.59:0/87010011 shutdown_connections 2024-03-19T03:06:16.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.619+0000 7fbac0bae700 1 --2- 172.21.15.59:0/87010011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbabc078d20 0x7fbabc077220 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:16.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.619+0000 7fbac0bae700 1 -- 172.21.15.59:0/87010011 >> 172.21.15.59:0/87010011 conn(0x7fbabc0fdf10 msgr2=0x7fbabc100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:16.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.619+0000 7fbac0bae700 1 -- 172.21.15.59:0/87010011 shutdown_connections 2024-03-19T03:06:16.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.619+0000 7fbac0bae700 1 -- 172.21.15.59:0/87010011 wait complete. 2024-03-19T03:06:16.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.620+0000 7fbac0bae700 1 Processor -- start 2024-03-19T03:06:16.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.620+0000 7fbac0bae700 1 -- start start 2024-03-19T03:06:16.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.620+0000 7fbac0bae700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbabc078d20 0x7fbabc112f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:16.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.620+0000 7fbac0bae700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc1134b0 con 0x7fbabc078d20 2024-03-19T03:06:16.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.620+0000 7fbaba59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbabc078d20 0x7fbabc112f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:16.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.620+0000 7fbaba59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbabc078d20 0x7fbabc112f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35894/0 (socket says 172.21.15.59:35894) 2024-03-19T03:06:16.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.620+0000 7fbaba59c700 1 -- 172.21.15.59:0/1899745089 learned_addr learned my addr 172.21.15.59:0/1899745089 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:16.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.620+0000 7fbaba59c700 1 -- 172.21.15.59:0/1899745089 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaac0094f0 con 0x7fbabc078d20 2024-03-19T03:06:16.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.620+0000 7fbaba59c700 1 --2- 172.21.15.59:0/1899745089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbabc078d20 0x7fbabc112f70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbaac009290 tx=0x7fbaac016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:16.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.621+0000 7fbaaaffd700 1 -- 172.21.15.59:0/1899745089 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbaac01c5c0 con 0x7fbabc078d20 2024-03-19T03:06:16.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.621+0000 7fbaaaffd700 1 -- 172.21.15.59:0/1899745089 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaac01c740 con 0x7fbabc078d20 2024-03-19T03:06:16.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.621+0000 7fbac0bae700 1 -- 172.21.15.59:0/1899745089 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbabc1136b0 con 0x7fbabc078d20 2024-03-19T03:06:16.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.621+0000 7fbaaaffd700 1 -- 172.21.15.59:0/1899745089 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbaac01ca50 con 0x7fbabc078d20 2024-03-19T03:06:16.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.621+0000 7fbac0bae700 1 -- 172.21.15.59:0/1899745089 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbabc114220 con 0x7fbabc078d20 2024-03-19T03:06:16.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.621+0000 7fbaaaffd700 1 -- 172.21.15.59:0/1899745089 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fbaac01a450 con 0x7fbabc078d20 2024-03-19T03:06:16.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.621+0000 7fbaaaffd700 1 -- 172.21.15.59:0/1899745089 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fbaac02e790 con 0x7fbabc078d20 2024-03-19T03:06:16.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.622+0000 7fbac0bae700 1 -- 172.21.15.59:0/1899745089 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbabc113b40 con 0x7fbabc078d20 2024-03-19T03:06:16.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.624+0000 7fbaaaffd700 1 -- 172.21.15.59:0/1899745089 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fbaac01ce30 con 0x7fbabc078d20 2024-03-19T03:06:16.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.696+0000 7fbac0bae700 1 -- 172.21.15.59:0/1899745089 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fbabc114c30 con 0x7fbabc078d20 2024-03-19T03:06:16.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.697+0000 7fbaaaffd700 1 -- 172.21.15.59:0/1899745089 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+324 (secure 0 0 0) 0x7fbaac01a890 con 0x7fbabc078d20 2024-03-19T03:06:16.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.697+0000 7fbac0bae700 1 -- 172.21.15.59:0/1899745089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbabc078d20 msgr2=0x7fbabc112f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:16.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.697+0000 7fbac0bae700 1 --2- 172.21.15.59:0/1899745089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbabc078d20 0x7fbabc112f70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbaac009290 tx=0x7fbaac016000 comp rx=0 tx=0).stop 2024-03-19T03:06:16.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.697+0000 7fbac0bae700 1 -- 172.21.15.59:0/1899745089 shutdown_connections 2024-03-19T03:06:16.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.697+0000 7fbac0bae700 1 --2- 172.21.15.59:0/1899745089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbabc078d20 0x7fbabc112f70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:16.337 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.697+0000 7fbac0bae700 1 -- 172.21.15.59:0/1899745089 >> 172.21.15.59:0/1899745089 conn(0x7fbabc0fdf10 msgr2=0x7fbabc109d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:16.337 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.698+0000 7fbac0bae700 1 -- 172.21.15.59:0/1899745089 shutdown_connections 2024-03-19T03:06:16.337 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:15.698+0000 7fbac0bae700 1 -- 172.21.15.59:0/1899745089 wait complete. 2024-03-19T03:06:16.337 INFO:teuthology.orchestra.run.smithi059.stdout:mon is available 2024-03-19T03:06:16.337 INFO:teuthology.orchestra.run.smithi059.stdout:Assimilating anything we can from ceph.conf... 2024-03-19T03:06:17.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:17 smithi059 ceph-mon[28750]: from='client.? 172.21.15.59:0/2354079342' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-19T03:06:17.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:17 smithi059 ceph-mon[28750]: from='client.? 172.21.15.59:0/2354079342' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-19T03:06:17.428 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-03-19T03:06:17.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout [global] 2024-03-19T03:06:17.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout fsid = 8c8ae530-e59d-11ee-95c9-87774f69a715 2024-03-19T03:06:17.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.59:3300,v1:172.21.15.59:6789] 2024-03-19T03:06:17.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-19T03:06:17.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-19T03:06:17.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-19T03:06:17.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-19T03:06:17.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-03-19T03:06:17.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-19T03:06:17.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-19T03:06:17.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-03-19T03:06:17.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout [osd] 2024-03-19T03:06:17.430 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-19T03:06:17.430 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-19T03:06:17.430 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-19T03:06:17.430 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-19T03:06:17.430 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-19T03:06:17.430 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.768+0000 7f523205b700 1 Processor -- start 2024-03-19T03:06:17.430 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.768+0000 7f523205b700 1 -- start start 2024-03-19T03:06:17.430 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.768+0000 7f523205b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f522c102720 0x7f522c102b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:17.430 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.768+0000 7f523205b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522c103070 con 0x7f522c102720 2024-03-19T03:06:17.430 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.768+0000 7f522b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f522c102720 0x7f522c102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:17.430 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.769+0000 7f522b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f522c102720 0x7f522c102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35904/0 (socket says 172.21.15.59:35904) 2024-03-19T03:06:17.430 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.769+0000 7f522b7fe700 1 -- 172.21.15.59:0/25554226 learned_addr learned my addr 172.21.15.59:0/25554226 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:17.430 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.769+0000 7f522b7fe700 1 -- 172.21.15.59:0/25554226 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522c1031b0 con 0x7f522c102720 2024-03-19T03:06:17.430 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.769+0000 7f522b7fe700 1 --2- 172.21.15.59:0/25554226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f522c102720 0x7f522c102b30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f521c009e50 tx=0x7f521c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed3381b86c48ae47 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:17.431 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.769+0000 7f522a7fc700 1 -- 172.21.15.59:0/25554226 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f521c0092e0 con 0x7f522c102720 2024-03-19T03:06:17.431 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.769+0000 7f522a7fc700 1 -- 172.21.15.59:0/25554226 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521c00ba20 con 0x7f522c102720 2024-03-19T03:06:17.431 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.769+0000 7f522a7fc700 1 -- 172.21.15.59:0/25554226 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f521c00bd30 con 0x7f522c102720 2024-03-19T03:06:17.431 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.770+0000 7f523205b700 1 -- 172.21.15.59:0/25554226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f522c102720 msgr2=0x7f522c102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:17.431 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.770+0000 7f523205b700 1 --2- 172.21.15.59:0/25554226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f522c102720 0x7f522c102b30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f521c009e50 tx=0x7f521c009840 comp rx=0 tx=0).stop 2024-03-19T03:06:17.431 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.770+0000 7f523205b700 1 -- 172.21.15.59:0/25554226 shutdown_connections 2024-03-19T03:06:17.431 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.770+0000 7f523205b700 1 --2- 172.21.15.59:0/25554226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f522c102720 0x7f522c102b30 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.431 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.770+0000 7f523205b700 1 -- 172.21.15.59:0/25554226 >> 172.21.15.59:0/25554226 conn(0x7f522c0fde80 msgr2=0x7f522c1002b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:17.431 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.770+0000 7f523205b700 1 -- 172.21.15.59:0/25554226 shutdown_connections 2024-03-19T03:06:17.431 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.770+0000 7f523205b700 1 -- 172.21.15.59:0/25554226 wait complete. 2024-03-19T03:06:17.431 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.770+0000 7f523205b700 1 Processor -- start 2024-03-19T03:06:17.431 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.770+0000 7f523205b700 1 -- start start 2024-03-19T03:06:17.431 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.771+0000 7f523205b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f522c102720 0x7f522c110e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:17.431 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.771+0000 7f523205b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522c111340 con 0x7f522c102720 2024-03-19T03:06:17.432 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.771+0000 7f522b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f522c102720 0x7f522c110e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:17.432 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.771+0000 7f522b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f522c102720 0x7f522c110e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35908/0 (socket says 172.21.15.59:35908) 2024-03-19T03:06:17.432 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.771+0000 7f522b7fe700 1 -- 172.21.15.59:0/2354079342 learned_addr learned my addr 172.21.15.59:0/2354079342 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:17.432 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.771+0000 7f522b7fe700 1 -- 172.21.15.59:0/2354079342 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c009ae0 con 0x7f522c102720 2024-03-19T03:06:17.432 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.771+0000 7f522b7fe700 1 --2- 172.21.15.59:0/2354079342 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f522c102720 0x7f522c110e00 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f522c103460 tx=0x7f521c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:17.432 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.771+0000 7f5223fff700 1 -- 172.21.15.59:0/2354079342 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f521c012b90 con 0x7f522c102720 2024-03-19T03:06:17.432 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.771+0000 7f5223fff700 1 -- 172.21.15.59:0/2354079342 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521c012d10 con 0x7f522c102720 2024-03-19T03:06:17.432 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.771+0000 7f5223fff700 1 -- 172.21.15.59:0/2354079342 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f521c01e400 con 0x7f522c102720 2024-03-19T03:06:17.432 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.771+0000 7f523205b700 1 -- 172.21.15.59:0/2354079342 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f522c111540 con 0x7f522c102720 2024-03-19T03:06:17.432 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.771+0000 7f523205b700 1 -- 172.21.15.59:0/2354079342 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f522c111f70 con 0x7f522c102720 2024-03-19T03:06:17.432 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.772+0000 7f5223fff700 1 -- 172.21.15.59:0/2354079342 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f521c019030 con 0x7f522c102720 2024-03-19T03:06:17.432 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.772+0000 7f5223fff700 1 -- 172.21.15.59:0/2354079342 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f521c01ed00 con 0x7f522c102720 2024-03-19T03:06:17.432 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.772+0000 7f523205b700 1 -- 172.21.15.59:0/2354079342 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f522c05f5f0 con 0x7f522c102720 2024-03-19T03:06:17.432 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.775+0000 7f5223fff700 1 -- 172.21.15.59:0/2354079342 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f521c017070 con 0x7f522c102720 2024-03-19T03:06:17.433 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.844+0000 7f523205b700 1 -- 172.21.15.59:0/2354079342 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f522c10aab0 con 0x7f522c102720 2024-03-19T03:06:17.433 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.847+0000 7f5223fff700 1 -- 172.21.15.59:0/2354079342 <== mon.0 v2:172.21.15.59:3300/0 7 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f521c038400 con 0x7f522c102720 2024-03-19T03:06:17.433 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.847+0000 7f5223fff700 1 -- 172.21.15.59:0/2354079342 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f521c026400 con 0x7f522c102720 2024-03-19T03:06:17.433 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.848+0000 7f523205b700 1 -- 172.21.15.59:0/2354079342 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f522c102720 msgr2=0x7f522c110e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:17.433 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.848+0000 7f523205b700 1 --2- 172.21.15.59:0/2354079342 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f522c102720 0x7f522c110e00 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f522c103460 tx=0x7f521c017000 comp rx=0 tx=0).stop 2024-03-19T03:06:17.433 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.848+0000 7f523205b700 1 -- 172.21.15.59:0/2354079342 shutdown_connections 2024-03-19T03:06:17.433 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.848+0000 7f523205b700 1 --2- 172.21.15.59:0/2354079342 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f522c102720 0x7f522c110e00 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:17.433 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.848+0000 7f523205b700 1 -- 172.21.15.59:0/2354079342 >> 172.21.15.59:0/2354079342 conn(0x7f522c0fde80 msgr2=0x7f522c107c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:17.433 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.848+0000 7f523205b700 1 -- 172.21.15.59:0/2354079342 shutdown_connections 2024-03-19T03:06:17.433 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:16.848+0000 7f523205b700 1 -- 172.21.15.59:0/2354079342 wait complete. 2024-03-19T03:06:17.433 INFO:teuthology.orchestra.run.smithi059.stdout:Generating new minimal ceph.conf... 2024-03-19T03:06:18.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:18 smithi059 ceph-mon[28750]: from='client.? 172.21.15.59:0/3868998987' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:06:18.533 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.845+0000 7ff546655700 1 Processor -- start 2024-03-19T03:06:18.533 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.845+0000 7ff546655700 1 -- start start 2024-03-19T03:06:18.533 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.845+0000 7ff546655700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff540076b70 0x7ff5401077f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:18.533 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.845+0000 7ff546655700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5401031e0 con 0x7ff540076b70 2024-03-19T03:06:18.533 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.845+0000 7ff53ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff540076b70 0x7ff5401077f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:18.533 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.846+0000 7ff53ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff540076b70 0x7ff5401077f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35924/0 (socket says 172.21.15.59:35924) 2024-03-19T03:06:18.534 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.846+0000 7ff53ffff700 1 -- 172.21.15.59:0/838837828 learned_addr learned my addr 172.21.15.59:0/838837828 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:18.534 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.846+0000 7ff53ffff700 1 -- 172.21.15.59:0/838837828 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff540107d30 con 0x7ff540076b70 2024-03-19T03:06:18.534 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.846+0000 7ff53ffff700 1 --2- 172.21.15.59:0/838837828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff540076b70 0x7ff5401077f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff5300098e0 tx=0x7ff530009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a04e95a28d63702 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:18.534 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.846+0000 7ff53effd700 1 -- 172.21.15.59:0/838837828 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff53000b860 con 0x7ff540076b70 2024-03-19T03:06:18.534 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.846+0000 7ff53effd700 1 -- 172.21.15.59:0/838837828 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff53000be50 con 0x7ff540076b70 2024-03-19T03:06:18.534 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.846+0000 7ff53effd700 1 -- 172.21.15.59:0/838837828 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff530010380 con 0x7ff540076b70 2024-03-19T03:06:18.534 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.847+0000 7ff546655700 1 -- 172.21.15.59:0/838837828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff540076b70 msgr2=0x7ff5401077f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:18.534 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.847+0000 7ff546655700 1 --2- 172.21.15.59:0/838837828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff540076b70 0x7ff5401077f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff5300098e0 tx=0x7ff530009250 comp rx=0 tx=0).stop 2024-03-19T03:06:18.534 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.847+0000 7ff546655700 1 -- 172.21.15.59:0/838837828 shutdown_connections 2024-03-19T03:06:18.534 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.847+0000 7ff546655700 1 --2- 172.21.15.59:0/838837828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff540076b70 0x7ff5401077f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.534 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.847+0000 7ff546655700 1 -- 172.21.15.59:0/838837828 >> 172.21.15.59:0/838837828 conn(0x7ff5400fce40 msgr2=0x7ff5400ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:18.534 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.847+0000 7ff546655700 1 -- 172.21.15.59:0/838837828 shutdown_connections 2024-03-19T03:06:18.534 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.847+0000 7ff546655700 1 -- 172.21.15.59:0/838837828 wait complete. 2024-03-19T03:06:18.535 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.847+0000 7ff546655700 1 Processor -- start 2024-03-19T03:06:18.535 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.847+0000 7ff546655700 1 -- start start 2024-03-19T03:06:18.535 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.848+0000 7ff546655700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff540076b70 0x7ff5401137d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:18.535 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.848+0000 7ff546655700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff540113d10 con 0x7ff540076b70 2024-03-19T03:06:18.535 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.848+0000 7ff53ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff540076b70 0x7ff5401137d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:18.535 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.848+0000 7ff53ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff540076b70 0x7ff5401137d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35934/0 (socket says 172.21.15.59:35934) 2024-03-19T03:06:18.535 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.848+0000 7ff53ffff700 1 -- 172.21.15.59:0/3868998987 learned_addr learned my addr 172.21.15.59:0/3868998987 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:18.535 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.848+0000 7ff53ffff700 1 -- 172.21.15.59:0/3868998987 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5300094f0 con 0x7ff540076b70 2024-03-19T03:06:18.535 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.848+0000 7ff53ffff700 1 --2- 172.21.15.59:0/3868998987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff540076b70 0x7ff5401137d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff5300090b0 tx=0x7ff530009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:18.535 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.848+0000 7ff53cff9700 1 -- 172.21.15.59:0/3868998987 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff530010760 con 0x7ff540076b70 2024-03-19T03:06:18.535 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.848+0000 7ff53cff9700 1 -- 172.21.15.59:0/3868998987 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff530010d50 con 0x7ff540076b70 2024-03-19T03:06:18.535 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.848+0000 7ff546655700 1 -- 172.21.15.59:0/3868998987 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff540113f10 con 0x7ff540076b70 2024-03-19T03:06:18.535 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.848+0000 7ff53cff9700 1 -- 172.21.15.59:0/3868998987 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff53001ace0 con 0x7ff540076b70 2024-03-19T03:06:18.535 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.848+0000 7ff546655700 1 -- 172.21.15.59:0/3868998987 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff540114a80 con 0x7ff540076b70 2024-03-19T03:06:18.537 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.849+0000 7ff53cff9700 1 -- 172.21.15.59:0/3868998987 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7ff53002d670 con 0x7ff540076b70 2024-03-19T03:06:18.537 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.849+0000 7ff53cff9700 1 -- 172.21.15.59:0/3868998987 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff530024780 con 0x7ff540076b70 2024-03-19T03:06:18.537 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.849+0000 7ff546655700 1 -- 172.21.15.59:0/3868998987 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff54005f5f0 con 0x7ff540076b70 2024-03-19T03:06:18.537 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.852+0000 7ff53cff9700 1 -- 172.21.15.59:0/3868998987 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7ff530029070 con 0x7ff540076b70 2024-03-19T03:06:18.538 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.921+0000 7ff546655700 1 -- 172.21.15.59:0/3868998987 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7ff5401089f0 con 0x7ff540076b70 2024-03-19T03:06:18.538 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.922+0000 7ff53cff9700 1 -- 172.21.15.59:0/3868998987 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7ff5300108c0 con 0x7ff540076b70 2024-03-19T03:06:18.538 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.922+0000 7ff546655700 1 -- 172.21.15.59:0/3868998987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff540076b70 msgr2=0x7ff5401137d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:18.538 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.922+0000 7ff546655700 1 --2- 172.21.15.59:0/3868998987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff540076b70 0x7ff5401137d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff5300090b0 tx=0x7ff530009250 comp rx=0 tx=0).stop 2024-03-19T03:06:18.538 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.922+0000 7ff546655700 1 -- 172.21.15.59:0/3868998987 shutdown_connections 2024-03-19T03:06:18.538 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.922+0000 7ff546655700 1 --2- 172.21.15.59:0/3868998987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff540076b70 0x7ff5401137d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:18.538 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.923+0000 7ff546655700 1 -- 172.21.15.59:0/3868998987 >> 172.21.15.59:0/3868998987 conn(0x7ff5400fce40 msgr2=0x7ff5400fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:18.538 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.923+0000 7ff546655700 1 -- 172.21.15.59:0/3868998987 shutdown_connections 2024-03-19T03:06:18.538 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:17.923+0000 7ff546655700 1 -- 172.21.15.59:0/3868998987 wait complete. 2024-03-19T03:06:18.538 INFO:teuthology.orchestra.run.smithi059.stdout:Restarting the monitor... 2024-03-19T03:06:18.788 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:18 smithi059 systemd[1]: Stopping Ceph mon.smithi059 for 8c8ae530-e59d-11ee-95c9-87774f69a715... 2024-03-19T03:06:19.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[28746]: 2024-03-19T03:06:18.786+0000 7fa4187b2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi059 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:06:19.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[28746]: 2024-03-19T03:06:18.786+0000 7fa4187b2700 -1 mon.smithi059@0(leader) e1 *** Got Signal Terminated *** 2024-03-19T03:06:19.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:18 smithi059 podman[29008]: 2024-03-19 03:06:18.899382212 +0000 UTC m=+0.208092776 container died 9449ca80148e10c531825073881fe12d1ab9836ea593daa4b2f11bc74100075b (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=quincy-b4843bd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-19T03:06:19.521 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:19 smithi059 podman[29008]: 2024-03-19 03:06:19.253252889 +0000 UTC m=+0.561963452 container cleanup 9449ca80148e10c531825073881fe12d1ab9836ea593daa4b2f11bc74100075b (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=quincy-b4843bd, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:06:19.521 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:19 smithi059 bash[29008]: ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059 2024-03-19T03:06:19.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:19 smithi059 podman[29022]: 2024-03-19 03:06:19.520793905 +0000 UTC m=+0.625869004 container remove 9449ca80148e10c531825073881fe12d1ab9836ea593daa4b2f11bc74100075b (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=quincy-b4843bd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True) 2024-03-19T03:06:19.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:19 smithi059 systemd[1]: ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@mon.smithi059.service: Deactivated successfully. 2024-03-19T03:06:19.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:19 smithi059 systemd[1]: Stopped Ceph mon.smithi059 for 8c8ae530-e59d-11ee-95c9-87774f69a715. 2024-03-19T03:06:19.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:19 smithi059 systemd[1]: Starting Ceph mon.smithi059 for 8c8ae530-e59d-11ee-95c9-87774f69a715... 2024-03-19T03:06:20.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:19 smithi059 podman[29115]: 2024-03-19 03:06:19.741125208 +0000 UTC m=+0.018002970 image pull 810955603526f1a5056294fef84995d43fccb8e7018e3b8031db5047002886ab quay.ceph.io/ceph-ci/ceph:quincy 2024-03-19T03:06:20.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:19 smithi059 podman[29115]: 2024-03-19 03:06:19.838326057 +0000 UTC m=+0.115203860 container create c8bc018eaa37f1af8b44320ee4ad903acd05b4c8deca961bf0ad1bf7e62df07d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=quincy-b4843bd, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-19T03:06:20.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 podman[29115]: 2024-03-19 03:06:20.048982958 +0000 UTC m=+0.325860720 container init c8bc018eaa37f1af8b44320ee4ad903acd05b4c8deca961bf0ad1bf7e62df07d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=quincy-b4843bd) 2024-03-19T03:06:20.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 podman[29115]: 2024-03-19 03:06:20.051809106 +0000 UTC m=+0.328686871 container start c8bc018eaa37f1af8b44320ee4ad903acd05b4c8deca961bf0ad1bf7e62df07d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059, RELEASE=quincy-b4843bd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-19T03:06:20.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: set uid:gid to 167:167 (ceph:ceph) 2024-03-19T03:06:20.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: ceph version 17.2.7-700-gb4843bd7 (b4843bd759866f9ee1bdfb1dfa084ec7d085b910) quincy (stable), process ceph-mon, pid 2 2024-03-19T03:06:20.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: pidfile_write: ignore empty --pid-file 2024-03-19T03:06:20.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: load: jerasure load: lrc 2024-03-19T03:06:20.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: RocksDB version: 6.15.5 2024-03-19T03:06:20.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-19T03:06:20.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Compile date Mar 18 2024 2024-03-19T03:06:20.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: DB SUMMARY 2024-03-19T03:06:20.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: DB Session ID: 0BPU7FYWU0TXBI54CT09 2024-03-19T03:06:20.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: CURRENT file: CURRENT 2024-03-19T03:06:20.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: IDENTITY file: IDENTITY 2024-03-19T03:06:20.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-03-19T03:06:20.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi059/store.db dir, Total Num: 1, files: 000008.sst 2024-03-19T03:06:20.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi059/store.db: 000010.log size: 97282 ; 2024-03-19T03:06:20.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.error_if_exists: 0 2024-03-19T03:06:20.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.create_if_missing: 0 2024-03-19T03:06:20.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.paranoid_checks: 1 2024-03-19T03:06:20.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-19T03:06:20.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.env: 0x5617d1cdda80 2024-03-19T03:06:20.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.fs: Posix File System 2024-03-19T03:06:20.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.info_log: 0x5617d400bc00 2024-03-19T03:06:20.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_file_opening_threads: 16 2024-03-19T03:06:20.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.statistics: (nil) 2024-03-19T03:06:20.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.use_fsync: 0 2024-03-19T03:06:20.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_log_file_size: 0 2024-03-19T03:06:20.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-19T03:06:20.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-19T03:06:20.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.keep_log_file_num: 1000 2024-03-19T03:06:20.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.recycle_log_file_num: 0 2024-03-19T03:06:20.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.allow_fallocate: 1 2024-03-19T03:06:20.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.allow_mmap_reads: 0 2024-03-19T03:06:20.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.allow_mmap_writes: 0 2024-03-19T03:06:20.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.use_direct_reads: 0 2024-03-19T03:06:20.093 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-19T03:06:20.093 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.create_missing_column_families: 0 2024-03-19T03:06:20.093 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.db_log_dir: 2024-03-19T03:06:20.093 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi059/store.db 2024-03-19T03:06:20.093 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-19T03:06:20.093 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-19T03:06:20.093 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-19T03:06:20.093 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-19T03:06:20.093 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-19T03:06:20.093 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-19T03:06:20.093 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.advise_random_on_open: 1 2024-03-19T03:06:20.093 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.db_write_buffer_size: 0 2024-03-19T03:06:20.093 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.write_buffer_manager: 0x5617d4d80ab0 2024-03-19T03:06:20.093 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-19T03:06:20.093 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-19T03:06:20.094 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-19T03:06:20.094 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-19T03:06:20.094 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.rate_limiter: (nil) 2024-03-19T03:06:20.094 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-19T03:06:20.094 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.wal_recovery_mode: 2 2024-03-19T03:06:20.094 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.enable_thread_tracking: 0 2024-03-19T03:06:20.094 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.enable_pipelined_write: 0 2024-03-19T03:06:20.095 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.unordered_write: 0 2024-03-19T03:06:20.096 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-19T03:06:20.096 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-19T03:06:20.096 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-19T03:06:20.096 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-19T03:06:20.096 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.row_cache: None 2024-03-19T03:06:20.096 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.wal_filter: None 2024-03-19T03:06:20.096 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-19T03:06:20.096 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.allow_ingest_behind: 0 2024-03-19T03:06:20.096 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.preserve_deletes: 0 2024-03-19T03:06:20.096 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.two_write_queues: 0 2024-03-19T03:06:20.096 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.manual_wal_flush: 0 2024-03-19T03:06:20.096 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.atomic_flush: 0 2024-03-19T03:06:20.096 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-19T03:06:20.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-19T03:06:20.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-19T03:06:20.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.log_readahead_size: 0 2024-03-19T03:06:20.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-19T03:06:20.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.best_efforts_recovery: 0 2024-03-19T03:06:20.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-19T03:06:20.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-19T03:06:20.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.allow_data_in_errors: 0 2024-03-19T03:06:20.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.db_host_id: __hostname__ 2024-03-19T03:06:20.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_background_jobs: 2 2024-03-19T03:06:20.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_background_compactions: -1 2024-03-19T03:06:20.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_subcompactions: 1 2024-03-19T03:06:20.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-19T03:06:20.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-19T03:06:20.098 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-19T03:06:20.098 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_total_wal_size: 0 2024-03-19T03:06:20.098 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-19T03:06:20.098 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-19T03:06:20.098 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-19T03:06:20.098 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-19T03:06:20.098 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_open_files: -1 2024-03-19T03:06:20.098 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.bytes_per_sync: 0 2024-03-19T03:06:20.098 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-19T03:06:20.098 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-19T03:06:20.098 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.compaction_readahead_size: 0 2024-03-19T03:06:20.098 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_background_flushes: -1 2024-03-19T03:06:20.098 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Compression algorithms supported: 2024-03-19T03:06:20.098 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-19T03:06:20.098 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: kZSTD supported: 0 2024-03-19T03:06:20.099 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: kXpressCompression supported: 0 2024-03-19T03:06:20.099 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: kLZ4HCCompression supported: 1 2024-03-19T03:06:20.099 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: kZlibCompression supported: 1 2024-03-19T03:06:20.099 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: kSnappyCompression supported: 1 2024-03-19T03:06:20.099 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: kLZ4Compression supported: 1 2024-03-19T03:06:20.099 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: kBZip2Compression supported: 0 2024-03-19T03:06:20.099 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-19T03:06:20.099 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi059/store.db/MANIFEST-000009 2024-03-19T03:06:20.099 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-03-19T03:06:20.099 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-19T03:06:20.099 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.merge_operator: 2024-03-19T03:06:20.099 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.compaction_filter: None 2024-03-19T03:06:20.099 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.compaction_filter_factory: None 2024-03-19T03:06:20.099 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.sst_partitioner_factory: None 2024-03-19T03:06:20.100 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-19T03:06:20.100 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-19T03:06:20.100 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5617d4060f58) 2024-03-19T03:06:20.100 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: cache_index_and_filter_blocks: 1 2024-03-19T03:06:20.100 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-19T03:06:20.100 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-19T03:06:20.100 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: pin_top_level_index_and_filter: 1 2024-03-19T03:06:20.100 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: index_type: 0 2024-03-19T03:06:20.100 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: data_block_index_type: 0 2024-03-19T03:06:20.100 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: index_shortening: 1 2024-03-19T03:06:20.100 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-19T03:06:20.100 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: hash_index_allow_collision: 1 2024-03-19T03:06:20.100 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: checksum: 1 2024-03-19T03:06:20.101 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: no_block_cache: 0 2024-03-19T03:06:20.101 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_cache: 0x5617d4093090 2024-03-19T03:06:20.101 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_cache_name: BinnedLRUCache 2024-03-19T03:06:20.101 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_cache_options: 2024-03-19T03:06:20.101 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: capacity : 536870912 2024-03-19T03:06:20.101 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: num_shard_bits : 4 2024-03-19T03:06:20.101 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: strict_capacity_limit : 0 2024-03-19T03:06:20.101 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: high_pri_pool_ratio: 0.000 2024-03-19T03:06:20.101 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_cache_compressed: (nil) 2024-03-19T03:06:20.101 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: persistent_cache: (nil) 2024-03-19T03:06:20.101 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_size: 4096 2024-03-19T03:06:20.101 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_size_deviation: 10 2024-03-19T03:06:20.101 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_restart_interval: 16 2024-03-19T03:06:20.101 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: index_block_restart_interval: 1 2024-03-19T03:06:20.101 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: metadata_block_size: 4096 2024-03-19T03:06:20.102 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: partition_filters: 0 2024-03-19T03:06:20.102 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: use_delta_encoding: 1 2024-03-19T03:06:20.102 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-19T03:06:20.102 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: whole_key_filtering: 1 2024-03-19T03:06:20.102 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: verify_compression: 0 2024-03-19T03:06:20.102 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: read_amp_bytes_per_bit: 0 2024-03-19T03:06:20.102 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: format_version: 4 2024-03-19T03:06:20.102 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: enable_index_compression: 1 2024-03-19T03:06:20.102 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_align: 0 2024-03-19T03:06:20.102 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.write_buffer_size: 33554432 2024-03-19T03:06:20.102 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_write_buffer_number: 2 2024-03-19T03:06:20.102 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.compression: NoCompression 2024-03-19T03:06:20.102 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.bottommost_compression: Disabled 2024-03-19T03:06:20.102 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.prefix_extractor: nullptr 2024-03-19T03:06:20.103 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-19T03:06:20.103 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.num_levels: 7 2024-03-19T03:06:20.103 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-19T03:06:20.103 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-19T03:06:20.103 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-19T03:06:20.103 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-19T03:06:20.103 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-19T03:06:20.103 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-19T03:06:20.103 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-19T03:06:20.103 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-19T03:06:20.103 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-19T03:06:20.103 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-19T03:06:20.103 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-19T03:06:20.103 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.compression_opts.level: 32767 2024-03-19T03:06:20.103 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.compression_opts.strategy: 0 2024-03-19T03:06:20.103 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-19T03:06:20.104 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-19T03:06:20.104 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-19T03:06:20.104 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.compression_opts.enabled: false 2024-03-19T03:06:20.104 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-19T03:06:20.104 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-19T03:06:20.104 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-19T03:06:20.104 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.target_file_size_base: 67108864 2024-03-19T03:06:20.104 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-19T03:06:20.104 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-19T03:06:20.104 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-19T03:06:20.104 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-19T03:06:20.104 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-19T03:06:20.104 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-19T03:06:20.104 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-19T03:06:20.104 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-19T03:06:20.105 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-19T03:06:20.105 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-19T03:06:20.105 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-19T03:06:20.105 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-19T03:06:20.105 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-19T03:06:20.105 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.arena_block_size: 4194304 2024-03-19T03:06:20.105 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-19T03:06:20.105 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-19T03:06:20.105 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-19T03:06:20.105 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.disable_auto_compactions: 0 2024-03-19T03:06:20.105 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-19T03:06:20.105 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-19T03:06:20.105 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-19T03:06:20.105 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-19T03:06:20.106 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-19T03:06:20.106 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-19T03:06:20.106 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-19T03:06:20.106 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-19T03:06:20.106 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-19T03:06:20.106 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-19T03:06:20.106 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.table_properties_collectors: 2024-03-19T03:06:20.106 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.inplace_update_support: 0 2024-03-19T03:06:20.106 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-19T03:06:20.106 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-19T03:06:20.106 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-19T03:06:20.106 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-19T03:06:20.106 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.bloom_locality: 0 2024-03-19T03:06:20.106 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.max_successive_merges: 0 2024-03-19T03:06:20.106 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-19T03:06:20.107 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.paranoid_file_checks: 0 2024-03-19T03:06:20.107 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.force_consistency_checks: 1 2024-03-19T03:06:20.107 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.report_bg_io_stats: 0 2024-03-19T03:06:20.107 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.ttl: 2592000 2024-03-19T03:06:20.107 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-19T03:06:20.107 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.enable_blob_files: false 2024-03-19T03:06:20.107 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.min_blob_size: 0 2024-03-19T03:06:20.107 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.blob_file_size: 268435456 2024-03-19T03:06:20.107 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-19T03:06:20.107 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-19T03:06:20.108 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-19T03:06:20.108 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi059/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-19T03:06:20.108 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-03-19T03:06:20.108 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-03-19T03:06:20.108 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710817580078604, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-03-19T03:06:20.108 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-03-19T03:06:20.108 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-19T03:06:20.108 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710817580079395, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92554, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90649, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13356, "raw_average_key_size": 49, "raw_value_size": 83364, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1710817580, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "8b42bd26-7312-4bb8-b6aa-ae7e5cca5060", "db_session_id": "0BPU7FYWU0TXBI54CT09"}} 2024-03-19T03:06:20.108 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-03-19T03:06:20.108 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710817580079708, "job": 1, "event": "recovery_finished"} 2024-03-19T03:06:20.108 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi059/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-19T03:06:20.108 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x5617d4027180 2024-03-19T03:06:20.108 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: DB pointer 0x5617d40fc000 2024-03-19T03:06:20.108 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-03-19T03:06:20.108 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-19T03:06:20.109 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ** DB Stats ** 2024-03-19T03:06:20.109 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-19T03:06:20.109 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-19T03:06:20.109 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-19T03:06:20.109 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-19T03:06:20.109 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-19T03:06:20.109 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-19T03:06:20.109 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-19T03:06:20.109 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: 2024-03-19T03:06:20.109 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ** Compaction Stats [default] ** 2024-03-19T03:06:20.109 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-19T03:06:20.109 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-19T03:06:20.109 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: L0 2/0 92.18 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.7 0.00 0.00 1 0.001 0 0 2024-03-19T03:06:20.109 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Sum 2/0 92.18 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.7 0.00 0.00 1 0.001 0 0 2024-03-19T03:06:20.110 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.7 0.00 0.00 1 0.001 0 0 2024-03-19T03:06:20.110 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: 2024-03-19T03:06:20.110 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ** Compaction Stats [default] ** 2024-03-19T03:06:20.110 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-19T03:06:20.110 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-19T03:06:20.110 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 161.7 0.00 0.00 1 0.001 0 0 2024-03-19T03:06:20.110 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-19T03:06:20.110 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-19T03:06:20.110 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-19T03:06:20.110 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-19T03:06:20.110 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-19T03:06:20.110 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-19T03:06:20.110 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Cumulative compaction: 0.00 GB write, 30.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T03:06:20.110 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Interval compaction: 0.00 GB write, 30.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T03:06:20.110 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-19T03:06:20.111 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: 2024-03-19T03:06:20.111 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-19T03:06:20.111 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: 2024-03-19T03:06:20.111 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ** Compaction Stats [default] ** 2024-03-19T03:06:20.111 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-19T03:06:20.111 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-19T03:06:20.111 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: L0 2/0 92.18 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.7 0.00 0.00 1 0.001 0 0 2024-03-19T03:06:20.111 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Sum 2/0 92.18 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.7 0.00 0.00 1 0.001 0 0 2024-03-19T03:06:20.111 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-19T03:06:20.111 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: 2024-03-19T03:06:20.111 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ** Compaction Stats [default] ** 2024-03-19T03:06:20.111 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-19T03:06:20.111 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-19T03:06:20.111 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 161.7 0.00 0.00 1 0.001 0 0 2024-03-19T03:06:20.112 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-19T03:06:20.112 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-19T03:06:20.112 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-19T03:06:20.112 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-19T03:06:20.112 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-19T03:06:20.112 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-19T03:06:20.112 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Cumulative compaction: 0.00 GB write, 29.75 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T03:06:20.112 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T03:06:20.112 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-19T03:06:20.112 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: 2024-03-19T03:06:20.112 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-19T03:06:20.112 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: starting mon.smithi059 rank 0 at public addrs [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] at bind addrs [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi059 fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 2024-03-19T03:06:20.112 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: mon.smithi059@-1(???) e1 preinit fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 2024-03-19T03:06:20.112 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: mon.smithi059@-1(???).mds e1 new map 2024-03-19T03:06:20.112 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: mon.smithi059@-1(???).mds e1 print_map 2024-03-19T03:06:20.113 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: e1 2024-03-19T03:06:20.113 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-19T03:06:20.113 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-19T03:06:20.113 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: legacy client fscid: -1 2024-03-19T03:06:20.113 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: 2024-03-19T03:06:20.113 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: No filesystems configured 2024-03-19T03:06:20.113 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: mon.smithi059@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-19T03:06:20.113 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: mon.smithi059@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T03:06:20.113 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: mon.smithi059@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T03:06:20.113 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: mon.smithi059@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T03:06:20.113 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: mon.smithi059@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-19T03:06:20.291 INFO:teuthology.orchestra.run.smithi059.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-19T03:06:20.409 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: mon.smithi059 is new leader, mons smithi059 in quorum (ranks 0) 2024-03-19T03:06:20.409 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: monmap e1: 1 mons at {smithi059=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-19T03:06:20.410 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: fsmap 2024-03-19T03:06:20.410 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: osdmap e1: 0 total, 0 up, 0 in 2024-03-19T03:06:20.410 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 ceph-mon[29133]: mgrmap e1: no daemons active 2024-03-19T03:06:20.410 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 bash[29115]: c8bc018eaa37f1af8b44320ee4ad903acd05b4c8deca961bf0ad1bf7e62df07d 2024-03-19T03:06:20.410 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:20 smithi059 systemd[1]: Started Ceph mon.smithi059 for 8c8ae530-e59d-11ee-95c9-87774f69a715. 2024-03-19T03:06:21.317 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.686+0000 7f8a65d20700 1 Processor -- start 2024-03-19T03:06:21.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.686+0000 7f8a65d20700 1 -- start start 2024-03-19T03:06:21.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.686+0000 7f8a65d20700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a60076be0 0x7f8a60105310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:21.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.686+0000 7f8a65d20700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a60105850 con 0x7f8a60076be0 2024-03-19T03:06:21.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.687+0000 7f8a5f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a60076be0 0x7f8a60105310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:21.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.687+0000 7f8a5f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a60076be0 0x7f8a60105310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35940/0 (socket says 172.21.15.59:35940) 2024-03-19T03:06:21.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.687+0000 7f8a5f7fe700 1 -- 172.21.15.59:0/3304715079 learned_addr learned my addr 172.21.15.59:0/3304715079 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:21.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.687+0000 7f8a5f7fe700 1 -- 172.21.15.59:0/3304715079 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a60105990 con 0x7f8a60076be0 2024-03-19T03:06:21.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.687+0000 7f8a5f7fe700 1 --2- 172.21.15.59:0/3304715079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a60076be0 0x7f8a60105310 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8a500098e0 tx=0x7f8a50009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a6fe8f4c8a0f5a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:21.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.687+0000 7f8a5e7fc700 1 -- 172.21.15.59:0/3304715079 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a5000b860 con 0x7f8a60076be0 2024-03-19T03:06:21.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.687+0000 7f8a5e7fc700 1 -- 172.21.15.59:0/3304715079 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f8a5000be50 con 0x7f8a60076be0 2024-03-19T03:06:21.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.687+0000 7f8a5e7fc700 1 -- 172.21.15.59:0/3304715079 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a50010380 con 0x7f8a60076be0 2024-03-19T03:06:21.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.688+0000 7f8a65d20700 1 -- 172.21.15.59:0/3304715079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a60076be0 msgr2=0x7f8a60105310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:21.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.688+0000 7f8a65d20700 1 --2- 172.21.15.59:0/3304715079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a60076be0 0x7f8a60105310 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8a500098e0 tx=0x7f8a50009250 comp rx=0 tx=0).stop 2024-03-19T03:06:21.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.688+0000 7f8a65d20700 1 -- 172.21.15.59:0/3304715079 shutdown_connections 2024-03-19T03:06:21.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.688+0000 7f8a65d20700 1 --2- 172.21.15.59:0/3304715079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a60076be0 0x7f8a60105310 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.688+0000 7f8a65d20700 1 -- 172.21.15.59:0/3304715079 >> 172.21.15.59:0/3304715079 conn(0x7f8a600fcdc0 msgr2=0x7f8a600ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:21.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.688+0000 7f8a65d20700 1 -- 172.21.15.59:0/3304715079 shutdown_connections 2024-03-19T03:06:21.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.688+0000 7f8a65d20700 1 -- 172.21.15.59:0/3304715079 wait complete. 2024-03-19T03:06:21.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.688+0000 7f8a65d20700 1 Processor -- start 2024-03-19T03:06:21.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.689+0000 7f8a65d20700 1 -- start start 2024-03-19T03:06:21.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.689+0000 7f8a65d20700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a60076be0 0x7f8a60102c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:21.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.689+0000 7f8a65d20700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a601031a0 con 0x7f8a60076be0 2024-03-19T03:06:21.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.689+0000 7f8a5f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a60076be0 0x7f8a60102c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:21.325 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.689+0000 7f8a5f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a60076be0 0x7f8a60102c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35948/0 (socket says 172.21.15.59:35948) 2024-03-19T03:06:21.325 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.689+0000 7f8a5f7fe700 1 -- 172.21.15.59:0/456831947 learned_addr learned my addr 172.21.15.59:0/456831947 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:21.325 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.689+0000 7f8a5f7fe700 1 -- 172.21.15.59:0/456831947 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a500094f0 con 0x7f8a60076be0 2024-03-19T03:06:21.325 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.689+0000 7f8a5f7fe700 1 --2- 172.21.15.59:0/456831947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a60076be0 0x7f8a60102c60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8a50003050 tx=0x7f8a50009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:21.325 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.689+0000 7f8a47fff700 1 -- 172.21.15.59:0/456831947 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a50010760 con 0x7f8a60076be0 2024-03-19T03:06:21.325 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.689+0000 7f8a47fff700 1 -- 172.21.15.59:0/456831947 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f8a50010d50 con 0x7f8a60076be0 2024-03-19T03:06:21.325 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.689+0000 7f8a65d20700 1 -- 172.21.15.59:0/456831947 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a60101210 con 0x7f8a60076be0 2024-03-19T03:06:21.325 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.690+0000 7f8a65d20700 1 -- 172.21.15.59:0/456831947 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a60101700 con 0x7f8a60076be0 2024-03-19T03:06:21.325 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.690+0000 7f8a47fff700 1 -- 172.21.15.59:0/456831947 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a5001ae20 con 0x7f8a60076be0 2024-03-19T03:06:21.325 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.690+0000 7f8a47fff700 1 -- 172.21.15.59:0/456831947 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f8a5002e6c0 con 0x7f8a60076be0 2024-03-19T03:06:21.325 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.690+0000 7f8a47fff700 1 -- 172.21.15.59:0/456831947 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8a50024a90 con 0x7f8a60076be0 2024-03-19T03:06:21.325 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.691+0000 7f8a65d20700 1 -- 172.21.15.59:0/456831947 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a6004bed0 con 0x7f8a60076be0 2024-03-19T03:06:21.325 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.693+0000 7f8a47fff700 1 -- 172.21.15.59:0/456831947 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f8a50029070 con 0x7f8a60076be0 2024-03-19T03:06:21.326 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.761+0000 7f8a65d20700 1 -- 172.21.15.59:0/456831947 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f8a60101940 con 0x7f8a60076be0 2024-03-19T03:06:21.326 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.764+0000 7f8a47fff700 1 -- 172.21.15.59:0/456831947 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f8a50006d70 con 0x7f8a60076be0 2024-03-19T03:06:21.326 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.764+0000 7f8a65d20700 1 -- 172.21.15.59:0/456831947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a60076be0 msgr2=0x7f8a60102c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:21.326 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.764+0000 7f8a65d20700 1 --2- 172.21.15.59:0/456831947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a60076be0 0x7f8a60102c60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8a50003050 tx=0x7f8a50009250 comp rx=0 tx=0).stop 2024-03-19T03:06:21.326 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.764+0000 7f8a65d20700 1 -- 172.21.15.59:0/456831947 shutdown_connections 2024-03-19T03:06:21.326 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.764+0000 7f8a65d20700 1 --2- 172.21.15.59:0/456831947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a60076be0 0x7f8a60102c60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:21.326 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.764+0000 7f8a65d20700 1 -- 172.21.15.59:0/456831947 >> 172.21.15.59:0/456831947 conn(0x7f8a600fcdc0 msgr2=0x7f8a600fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:21.326 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.764+0000 7f8a65d20700 1 -- 172.21.15.59:0/456831947 shutdown_connections 2024-03-19T03:06:21.326 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:20.764+0000 7f8a65d20700 1 -- 172.21.15.59:0/456831947 wait complete. 2024-03-19T03:06:21.326 INFO:teuthology.orchestra.run.smithi059.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-19T03:06:21.326 INFO:teuthology.orchestra.run.smithi059.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-19T03:06:21.326 INFO:teuthology.orchestra.run.smithi059.stdout:Creating mgr... 2024-03-19T03:06:21.326 INFO:teuthology.orchestra.run.smithi059.stdout:Verifying port 9283 ... 2024-03-19T03:06:21.496 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@mgr.smithi059.azpecc 2024-03-19T03:06:21.496 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@mgr.smithi059.azpecc.service: Unit ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@mgr.smithi059.azpecc.service not loaded. 2024-03-19T03:06:21.654 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8c8ae530-e59d-11ee-95c9-87774f69a715.target.wants/ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@mgr.smithi059.azpecc.service → /etc/systemd/system/ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@.service. 2024-03-19T03:06:21.999 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:21 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/456831947' entity='client.admin' 2024-03-19T03:06:22.340 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-19T03:06:22.341 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout disabled 2024-03-19T03:06:22.348 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-19T03:06:22.348 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout inactive 2024-03-19T03:06:22.348 INFO:teuthology.orchestra.run.smithi059.stdout:firewalld.service is not enabled 2024-03-19T03:06:22.348 INFO:teuthology.orchestra.run.smithi059.stdout:Not possible to enable service . firewalld.service is not available 2024-03-19T03:06:22.356 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-19T03:06:22.357 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout disabled 2024-03-19T03:06:22.364 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-19T03:06:22.364 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout inactive 2024-03-19T03:06:22.364 INFO:teuthology.orchestra.run.smithi059.stdout:firewalld.service is not enabled 2024-03-19T03:06:22.364 INFO:teuthology.orchestra.run.smithi059.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-03-19T03:06:22.364 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for mgr to start... 2024-03-19T03:06:22.365 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for mgr... 2024-03-19T03:06:23.174 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:22 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/770348715' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-19T03:06:23.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-03-19T03:06:23.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2024-03-19T03:06:23.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "fsid": "8c8ae530-e59d-11ee-95c9-87774f69a715", 2024-03-19T03:06:23.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "health": { 2024-03-19T03:06:23.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-19T03:06:23.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-19T03:06:23.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-19T03:06:23.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:23.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-19T03:06:23.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-19T03:06:23.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 0 2024-03-19T03:06:23.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:06:23.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-19T03:06:23.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "smithi059" 2024-03-19T03:06:23.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:06:23.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-19T03:06:23.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-19T03:06:23.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:06:23.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-19T03:06:23.465 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-19T03:06:23.465 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:23.465 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-19T03:06:23.465 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:06:23.465 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-19T03:06:23.465 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-19T03:06:23.465 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-19T03:06:23.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-19T03:06:23.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-19T03:06:23.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-19T03:06:23.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:23.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-19T03:06:23.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-19T03:06:23.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-19T03:06:23.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-19T03:06:23.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-19T03:06:23.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-19T03:06:23.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-19T03:06:23.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-19T03:06:23.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-19T03:06:23.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:23.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-19T03:06:23.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:06:23.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-19T03:06:23.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-19T03:06:23.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:23.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-19T03:06:23.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-19T03:06:23.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-19T03:06:23.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-19T03:06:23.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-19T03:06:23.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-19T03:06:23.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "restful" 2024-03-19T03:06:23.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:06:23.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T03:06:23.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:23.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-19T03:06:23.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:06:23.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "modified": "2024-03-19T03:06:15.064565+0000", 2024-03-19T03:06:23.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T03:06:23.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:23.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-19T03:06:23.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2024-03-19T03:06:23.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.793+0000 7f889cf83700 1 Processor -- start 2024-03-19T03:06:23.479 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.793+0000 7f889cf83700 1 -- start start 2024-03-19T03:06:23.479 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.793+0000 7f889cf83700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8898071320 0x7f8898107720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:23.479 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.793+0000 7f889cf83700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8898103110 con 0x7f8898071320 2024-03-19T03:06:23.479 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.794+0000 7f889659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8898071320 0x7f8898107720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:23.479 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.794+0000 7f889659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8898071320 0x7f8898107720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35982/0 (socket says 172.21.15.59:35982) 2024-03-19T03:06:23.479 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.794+0000 7f889659c700 1 -- 172.21.15.59:0/3305386121 learned_addr learned my addr 172.21.15.59:0/3305386121 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:23.479 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.794+0000 7f889659c700 1 -- 172.21.15.59:0/3305386121 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8898107c60 con 0x7f8898071320 2024-03-19T03:06:23.479 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.794+0000 7f889659c700 1 --2- 172.21.15.59:0/3305386121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8898071320 0x7f8898107720 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f88880098e0 tx=0x7f8888009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47526b0aca9aa04a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:23.479 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.794+0000 7f889559a700 1 -- 172.21.15.59:0/3305386121 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f888800b860 con 0x7f8898071320 2024-03-19T03:06:23.479 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.794+0000 7f889559a700 1 -- 172.21.15.59:0/3305386121 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f888800be50 con 0x7f8898071320 2024-03-19T03:06:23.479 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.794+0000 7f889559a700 1 -- 172.21.15.59:0/3305386121 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8888010380 con 0x7f8898071320 2024-03-19T03:06:23.479 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.795+0000 7f889cf83700 1 -- 172.21.15.59:0/3305386121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8898071320 msgr2=0x7f8898107720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:23.479 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.795+0000 7f889cf83700 1 --2- 172.21.15.59:0/3305386121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8898071320 0x7f8898107720 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f88880098e0 tx=0x7f8888009250 comp rx=0 tx=0).stop 2024-03-19T03:06:23.480 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.795+0000 7f889cf83700 1 -- 172.21.15.59:0/3305386121 shutdown_connections 2024-03-19T03:06:23.480 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.795+0000 7f889cf83700 1 --2- 172.21.15.59:0/3305386121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8898071320 0x7f8898107720 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:23.480 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.795+0000 7f889cf83700 1 -- 172.21.15.59:0/3305386121 >> 172.21.15.59:0/3305386121 conn(0x7f88980fcd50 msgr2=0x7f88980ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:23.480 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.795+0000 7f889cf83700 1 -- 172.21.15.59:0/3305386121 shutdown_connections 2024-03-19T03:06:23.480 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.795+0000 7f889cf83700 1 -- 172.21.15.59:0/3305386121 wait complete. 2024-03-19T03:06:23.480 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.796+0000 7f889cf83700 1 Processor -- start 2024-03-19T03:06:23.480 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.796+0000 7f889cf83700 1 -- start start 2024-03-19T03:06:23.480 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.796+0000 7f889cf83700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8898071320 0x7f8898113810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:23.480 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.796+0000 7f889cf83700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8898113d50 con 0x7f8898071320 2024-03-19T03:06:23.480 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.796+0000 7f889659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8898071320 0x7f8898113810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:23.480 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.796+0000 7f889659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8898071320 0x7f8898113810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35986/0 (socket says 172.21.15.59:35986) 2024-03-19T03:06:23.480 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.796+0000 7f889659c700 1 -- 172.21.15.59:0/770348715 learned_addr learned my addr 172.21.15.59:0/770348715 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:23.481 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.796+0000 7f889659c700 1 -- 172.21.15.59:0/770348715 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88880094f0 con 0x7f8898071320 2024-03-19T03:06:23.481 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.797+0000 7f889659c700 1 --2- 172.21.15.59:0/770348715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8898071320 0x7f8898113810 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f88880090b0 tx=0x7f8888009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:23.481 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.797+0000 7f8886ffd700 1 -- 172.21.15.59:0/770348715 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8888010760 con 0x7f8898071320 2024-03-19T03:06:23.481 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.797+0000 7f8886ffd700 1 -- 172.21.15.59:0/770348715 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f8888010d50 con 0x7f8898071320 2024-03-19T03:06:23.481 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.797+0000 7f889cf83700 1 -- 172.21.15.59:0/770348715 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8898113f50 con 0x7f8898071320 2024-03-19T03:06:23.481 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.797+0000 7f8886ffd700 1 -- 172.21.15.59:0/770348715 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f888801ace0 con 0x7f8898071320 2024-03-19T03:06:23.481 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.797+0000 7f889cf83700 1 -- 172.21.15.59:0/770348715 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8898114ac0 con 0x7f8898071320 2024-03-19T03:06:23.481 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.797+0000 7f8886ffd700 1 -- 172.21.15.59:0/770348715 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f888802d670 con 0x7f8898071320 2024-03-19T03:06:23.481 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.797+0000 7f8886ffd700 1 -- 172.21.15.59:0/770348715 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f88880247d0 con 0x7f8898071320 2024-03-19T03:06:23.481 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.798+0000 7f889cf83700 1 -- 172.21.15.59:0/770348715 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88981143e0 con 0x7f8898071320 2024-03-19T03:06:23.481 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.800+0000 7f8886ffd700 1 -- 172.21.15.59:0/770348715 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f8888029070 con 0x7f8898071320 2024-03-19T03:06:23.481 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.873+0000 7f889cf83700 1 -- 172.21.15.59:0/770348715 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f889805f5f0 con 0x7f8898071320 2024-03-19T03:06:23.481 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.873+0000 7f8886ffd700 1 -- 172.21.15.59:0/770348715 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f88880108c0 con 0x7f8898071320 2024-03-19T03:06:23.481 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.874+0000 7f889cf83700 1 -- 172.21.15.59:0/770348715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8898071320 msgr2=0x7f8898113810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:23.481 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.874+0000 7f889cf83700 1 --2- 172.21.15.59:0/770348715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8898071320 0x7f8898113810 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f88880090b0 tx=0x7f8888009250 comp rx=0 tx=0).stop 2024-03-19T03:06:23.482 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.874+0000 7f889cf83700 1 -- 172.21.15.59:0/770348715 shutdown_connections 2024-03-19T03:06:23.482 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.874+0000 7f889cf83700 1 --2- 172.21.15.59:0/770348715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8898071320 0x7f8898113810 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:23.482 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.874+0000 7f889cf83700 1 -- 172.21.15.59:0/770348715 >> 172.21.15.59:0/770348715 conn(0x7f88980fcd50 msgr2=0x7f88980fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:23.482 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.874+0000 7f889cf83700 1 -- 172.21.15.59:0/770348715 shutdown_connections 2024-03-19T03:06:23.482 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:22.874+0000 7f889cf83700 1 -- 172.21.15.59:0/770348715 wait complete. 2024-03-19T03:06:23.482 INFO:teuthology.orchestra.run.smithi059.stdout:mgr not available, waiting (1/15)... 2024-03-19T03:06:26.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:26 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/529074188' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-19T03:06:26.661 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-03-19T03:06:26.662 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2024-03-19T03:06:26.662 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "fsid": "8c8ae530-e59d-11ee-95c9-87774f69a715", 2024-03-19T03:06:26.662 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "health": { 2024-03-19T03:06:26.662 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-19T03:06:26.662 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-19T03:06:26.662 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-19T03:06:26.662 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:26.662 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-19T03:06:26.662 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-19T03:06:26.662 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 0 2024-03-19T03:06:26.663 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:06:26.663 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-19T03:06:26.663 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "smithi059" 2024-03-19T03:06:26.663 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:06:26.663 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-19T03:06:26.663 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-19T03:06:26.663 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:06:26.663 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-19T03:06:26.663 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-19T03:06:26.663 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:26.663 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-19T03:06:26.663 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:06:26.664 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-19T03:06:26.664 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-19T03:06:26.664 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-19T03:06:26.664 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-19T03:06:26.664 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-19T03:06:26.664 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-19T03:06:26.664 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:26.664 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-19T03:06:26.664 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-19T03:06:26.664 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-19T03:06:26.664 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-19T03:06:26.664 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-19T03:06:26.665 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-19T03:06:26.665 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-19T03:06:26.665 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-19T03:06:26.665 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-19T03:06:26.665 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:26.665 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-19T03:06:26.665 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:06:26.665 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-19T03:06:26.665 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-19T03:06:26.665 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:26.665 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-19T03:06:26.665 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-19T03:06:26.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-19T03:06:26.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-19T03:06:26.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-19T03:06:26.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-19T03:06:26.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "restful" 2024-03-19T03:06:26.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:06:26.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T03:06:26.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:26.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-19T03:06:26.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:06:26.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "modified": "2024-03-19T03:06:15.064565+0000", 2024-03-19T03:06:26.667 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T03:06:26.667 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:26.667 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-19T03:06:26.667 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2024-03-19T03:06:26.667 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.033+0000 7f1a27ad7700 1 Processor -- start 2024-03-19T03:06:26.667 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.034+0000 7f1a27ad7700 1 -- start start 2024-03-19T03:06:26.667 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.034+0000 7f1a27ad7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a20100ac0 0x7f1a20100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:26.667 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.034+0000 7f1a27ad7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a20101410 con 0x7f1a20100ac0 2024-03-19T03:06:26.667 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.034+0000 7f1a25873700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a20100ac0 0x7f1a20100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:26.667 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.034+0000 7f1a25873700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a20100ac0 0x7f1a20100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58414/0 (socket says 172.21.15.59:58414) 2024-03-19T03:06:26.667 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.034+0000 7f1a25873700 1 -- 172.21.15.59:0/4200640884 learned_addr learned my addr 172.21.15.59:0/4200640884 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:26.667 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.034+0000 7f1a25873700 1 -- 172.21.15.59:0/4200640884 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a20101550 con 0x7f1a20100ac0 2024-03-19T03:06:26.668 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.034+0000 7f1a25873700 1 --2- 172.21.15.59:0/4200640884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a20100ac0 0x7f1a20100ed0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c0098e0 tx=0x7f1a1c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7f62dc0e6966acd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:26.668 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.035+0000 7f1a24871700 1 -- 172.21.15.59:0/4200640884 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a1c00b860 con 0x7f1a20100ac0 2024-03-19T03:06:26.668 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.035+0000 7f1a24871700 1 -- 172.21.15.59:0/4200640884 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1a1c00be50 con 0x7f1a20100ac0 2024-03-19T03:06:26.668 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.035+0000 7f1a24871700 1 -- 172.21.15.59:0/4200640884 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a1c010380 con 0x7f1a20100ac0 2024-03-19T03:06:26.668 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.035+0000 7f1a27ad7700 1 -- 172.21.15.59:0/4200640884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a20100ac0 msgr2=0x7f1a20100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:26.668 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.035+0000 7f1a27ad7700 1 --2- 172.21.15.59:0/4200640884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a20100ac0 0x7f1a20100ed0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c0098e0 tx=0x7f1a1c009250 comp rx=0 tx=0).stop 2024-03-19T03:06:26.668 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.036+0000 7f1a27ad7700 1 -- 172.21.15.59:0/4200640884 shutdown_connections 2024-03-19T03:06:26.668 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.036+0000 7f1a27ad7700 1 --2- 172.21.15.59:0/4200640884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a20100ac0 0x7f1a20100ed0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:26.668 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.036+0000 7f1a27ad7700 1 -- 172.21.15.59:0/4200640884 >> 172.21.15.59:0/4200640884 conn(0x7f1a200fc500 msgr2=0x7f1a200fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:26.668 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.036+0000 7f1a27ad7700 1 -- 172.21.15.59:0/4200640884 shutdown_connections 2024-03-19T03:06:26.668 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.036+0000 7f1a27ad7700 1 -- 172.21.15.59:0/4200640884 wait complete. 2024-03-19T03:06:26.668 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.036+0000 7f1a27ad7700 1 Processor -- start 2024-03-19T03:06:26.668 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.036+0000 7f1a27ad7700 1 -- start start 2024-03-19T03:06:26.669 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.036+0000 7f1a27ad7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a20100ac0 0x7f1a2010f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:26.669 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.036+0000 7f1a27ad7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2010fa70 con 0x7f1a20100ac0 2024-03-19T03:06:26.669 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.036+0000 7f1a25873700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a20100ac0 0x7f1a2010f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:26.669 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.037+0000 7f1a25873700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a20100ac0 0x7f1a2010f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58430/0 (socket says 172.21.15.59:58430) 2024-03-19T03:06:26.669 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.037+0000 7f1a25873700 1 -- 172.21.15.59:0/529074188 learned_addr learned my addr 172.21.15.59:0/529074188 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:26.669 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.037+0000 7f1a25873700 1 -- 172.21.15.59:0/529074188 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1c0094f0 con 0x7f1a20100ac0 2024-03-19T03:06:26.669 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.037+0000 7f1a25873700 1 --2- 172.21.15.59:0/529074188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a20100ac0 0x7f1a2010f530 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c00b440 tx=0x7f1a1c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:26.669 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.037+0000 7f1a127fc700 1 -- 172.21.15.59:0/529074188 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a1c010760 con 0x7f1a20100ac0 2024-03-19T03:06:26.669 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.037+0000 7f1a127fc700 1 -- 172.21.15.59:0/529074188 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1a1c010d50 con 0x7f1a20100ac0 2024-03-19T03:06:26.669 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.037+0000 7f1a127fc700 1 -- 172.21.15.59:0/529074188 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a1c01ae20 con 0x7f1a20100ac0 2024-03-19T03:06:26.669 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.037+0000 7f1a27ad7700 1 -- 172.21.15.59:0/529074188 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a2010fc70 con 0x7f1a20100ac0 2024-03-19T03:06:26.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.037+0000 7f1a27ad7700 1 -- 172.21.15.59:0/529074188 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a201107e0 con 0x7f1a20100ac0 2024-03-19T03:06:26.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.037+0000 7f1a127fc700 1 -- 172.21.15.59:0/529074188 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f1a1c02e6c0 con 0x7f1a20100ac0 2024-03-19T03:06:26.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.038+0000 7f1a127fc700 1 -- 172.21.15.59:0/529074188 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1a1c024a80 con 0x7f1a20100ac0 2024-03-19T03:06:26.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.038+0000 7f1a27ad7700 1 -- 172.21.15.59:0/529074188 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a2005f5f0 con 0x7f1a20100ac0 2024-03-19T03:06:26.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.041+0000 7f1a127fc700 1 -- 172.21.15.59:0/529074188 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f1a1c029070 con 0x7f1a20100ac0 2024-03-19T03:06:26.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.114+0000 7f1a27ad7700 1 -- 172.21.15.59:0/529074188 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1a20063c70 con 0x7f1a20100ac0 2024-03-19T03:06:26.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.115+0000 7f1a127fc700 1 -- 172.21.15.59:0/529074188 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f1a1c01a6a0 con 0x7f1a20100ac0 2024-03-19T03:06:26.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.116+0000 7f1a27ad7700 1 -- 172.21.15.59:0/529074188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a20100ac0 msgr2=0x7f1a2010f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:26.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.116+0000 7f1a27ad7700 1 --2- 172.21.15.59:0/529074188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a20100ac0 0x7f1a2010f530 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c00b440 tx=0x7f1a1c009250 comp rx=0 tx=0).stop 2024-03-19T03:06:26.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.116+0000 7f1a27ad7700 1 -- 172.21.15.59:0/529074188 shutdown_connections 2024-03-19T03:06:26.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.116+0000 7f1a27ad7700 1 --2- 172.21.15.59:0/529074188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a20100ac0 0x7f1a2010f530 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:26.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.116+0000 7f1a27ad7700 1 -- 172.21.15.59:0/529074188 >> 172.21.15.59:0/529074188 conn(0x7f1a200fc500 msgr2=0x7f1a200fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:26.671 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.116+0000 7f1a27ad7700 1 -- 172.21.15.59:0/529074188 shutdown_connections 2024-03-19T03:06:26.671 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:26.116+0000 7f1a27ad7700 1 -- 172.21.15.59:0/529074188 wait complete. 2024-03-19T03:06:26.671 INFO:teuthology.orchestra.run.smithi059.stdout:mgr not available, waiting (2/15)... 2024-03-19T03:06:29.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:28 smithi059 ceph-mon[29133]: Activating manager daemon smithi059.azpecc 2024-03-19T03:06:29.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:28 smithi059 ceph-mon[29133]: mgrmap e2: smithi059.azpecc(active, starting, since 0.00278533s) 2024-03-19T03:06:29.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:28 smithi059 ceph-mon[29133]: from='mgr.14100 172.21.15.59:0/408639015' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T03:06:29.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:28 smithi059 ceph-mon[29133]: from='mgr.14100 172.21.15.59:0/408639015' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T03:06:29.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:28 smithi059 ceph-mon[29133]: from='mgr.14100 172.21.15.59:0/408639015' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T03:06:29.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:28 smithi059 ceph-mon[29133]: from='mgr.14100 172.21.15.59:0/408639015' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-03-19T03:06:29.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:28 smithi059 ceph-mon[29133]: from='mgr.14100 172.21.15.59:0/408639015' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mgr metadata", "who": "smithi059.azpecc", "id": "smithi059.azpecc"}]: dispatch 2024-03-19T03:06:29.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:28 smithi059 ceph-mon[29133]: Manager daemon smithi059.azpecc is now available 2024-03-19T03:06:29.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:28 smithi059 ceph-mon[29133]: from='mgr.14100 172.21.15.59:0/408639015' entity='mgr.smithi059.azpecc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.azpecc/mirror_snapshot_schedule"}]: dispatch 2024-03-19T03:06:29.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:28 smithi059 ceph-mon[29133]: from='mgr.14100 172.21.15.59:0/408639015' entity='mgr.smithi059.azpecc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.azpecc/trash_purge_schedule"}]: dispatch 2024-03-19T03:06:29.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:28 smithi059 ceph-mon[29133]: from='mgr.14100 172.21.15.59:0/408639015' entity='mgr.smithi059.azpecc' 2024-03-19T03:06:29.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:28 smithi059 ceph-mon[29133]: from='mgr.14100 172.21.15.59:0/408639015' entity='mgr.smithi059.azpecc' 2024-03-19T03:06:29.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:28 smithi059 ceph-mon[29133]: from='mgr.14100 172.21.15.59:0/408639015' entity='mgr.smithi059.azpecc' 2024-03-19T03:06:29.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-03-19T03:06:29.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2024-03-19T03:06:29.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "fsid": "8c8ae530-e59d-11ee-95c9-87774f69a715", 2024-03-19T03:06:29.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "health": { 2024-03-19T03:06:29.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-19T03:06:29.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-19T03:06:29.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-19T03:06:29.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:29.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-19T03:06:29.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-19T03:06:29.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 0 2024-03-19T03:06:29.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:06:29.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-19T03:06:29.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "smithi059" 2024-03-19T03:06:29.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:06:29.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-19T03:06:29.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-19T03:06:29.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:06:29.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-19T03:06:29.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-19T03:06:29.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:29.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-19T03:06:29.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:06:29.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-19T03:06:29.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-19T03:06:29.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-19T03:06:29.844 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-19T03:06:29.844 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-19T03:06:29.844 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-19T03:06:29.844 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:29.844 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-19T03:06:29.844 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-19T03:06:29.845 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-19T03:06:29.845 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-19T03:06:29.845 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-19T03:06:29.845 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-19T03:06:29.845 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-19T03:06:29.845 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-19T03:06:29.845 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-19T03:06:29.845 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:29.845 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-19T03:06:29.845 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:06:29.845 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-19T03:06:29.845 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-19T03:06:29.845 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:29.845 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-19T03:06:29.846 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-19T03:06:29.846 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-19T03:06:29.846 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-19T03:06:29.846 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-19T03:06:29.846 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-19T03:06:29.846 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "restful" 2024-03-19T03:06:29.846 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:06:29.846 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T03:06:29.846 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:29.846 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-19T03:06:29.846 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:06:29.846 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "modified": "2024-03-19T03:06:15.064565+0000", 2024-03-19T03:06:29.847 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T03:06:29.847 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:29.847 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-19T03:06:29.847 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2024-03-19T03:06:29.847 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.102+0000 7f045a4ee700 1 Processor -- start 2024-03-19T03:06:29.847 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.102+0000 7f045a4ee700 1 -- start start 2024-03-19T03:06:29.847 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.102+0000 7f045a4ee700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454102be0 0x7f0454102ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:29.847 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.102+0000 7f045a4ee700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0454103530 con 0x7f0454102be0 2024-03-19T03:06:29.847 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.102+0000 7f0453fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454102be0 0x7f0454102ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:29.847 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.103+0000 7f0453fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454102be0 0x7f0454102ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58510/0 (socket says 172.21.15.59:58510) 2024-03-19T03:06:29.847 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.103+0000 7f0453fff700 1 -- 172.21.15.59:0/1619593297 learned_addr learned my addr 172.21.15.59:0/1619593297 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:29.847 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.103+0000 7f0453fff700 1 -- 172.21.15.59:0/1619593297 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0454103670 con 0x7f0454102be0 2024-03-19T03:06:29.847 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.103+0000 7f0453fff700 1 --2- 172.21.15.59:0/1619593297 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454102be0 0x7f0454102ff0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f044400bb80 tx=0x7f044400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b19e4146d328f6e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:29.848 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.103+0000 7f0452ffd700 1 -- 172.21.15.59:0/1619593297 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f044400dd00 con 0x7f0454102be0 2024-03-19T03:06:29.848 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.103+0000 7f0452ffd700 1 -- 172.21.15.59:0/1619593297 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f044400de60 con 0x7f0454102be0 2024-03-19T03:06:29.848 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.103+0000 7f0452ffd700 1 -- 172.21.15.59:0/1619593297 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04440188a0 con 0x7f0454102be0 2024-03-19T03:06:29.848 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.104+0000 7f045a4ee700 1 -- 172.21.15.59:0/1619593297 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454102be0 msgr2=0x7f0454102ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:29.848 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.104+0000 7f045a4ee700 1 --2- 172.21.15.59:0/1619593297 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454102be0 0x7f0454102ff0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f044400bb80 tx=0x7f044400b4f0 comp rx=0 tx=0).stop 2024-03-19T03:06:29.848 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.104+0000 7f045a4ee700 1 -- 172.21.15.59:0/1619593297 shutdown_connections 2024-03-19T03:06:29.848 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.104+0000 7f045a4ee700 1 --2- 172.21.15.59:0/1619593297 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454102be0 0x7f0454102ff0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:29.848 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.104+0000 7f045a4ee700 1 -- 172.21.15.59:0/1619593297 >> 172.21.15.59:0/1619593297 conn(0x7f04540fc500 msgr2=0x7f04540fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:29.848 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.104+0000 7f045a4ee700 1 -- 172.21.15.59:0/1619593297 shutdown_connections 2024-03-19T03:06:29.848 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.104+0000 7f045a4ee700 1 -- 172.21.15.59:0/1619593297 wait complete. 2024-03-19T03:06:29.848 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.104+0000 7f045a4ee700 1 Processor -- start 2024-03-19T03:06:29.848 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.104+0000 7f045a4ee700 1 -- start start 2024-03-19T03:06:29.848 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.105+0000 7f045a4ee700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454102be0 0x7f045410f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:29.848 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.105+0000 7f045a4ee700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045410f830 con 0x7f0454102be0 2024-03-19T03:06:29.849 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.105+0000 7f0453fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454102be0 0x7f045410f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:29.849 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.105+0000 7f0453fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454102be0 0x7f045410f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58524/0 (socket says 172.21.15.59:58524) 2024-03-19T03:06:29.849 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.105+0000 7f0453fff700 1 -- 172.21.15.59:0/3587417207 learned_addr learned my addr 172.21.15.59:0/3587417207 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:29.849 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.105+0000 7f0453fff700 1 -- 172.21.15.59:0/3587417207 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044400b790 con 0x7f0454102be0 2024-03-19T03:06:29.849 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.105+0000 7f0453fff700 1 --2- 172.21.15.59:0/3587417207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454102be0 0x7f045410f2f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f044400d650 tx=0x7f044400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:29.849 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.105+0000 7f0450ff9700 1 -- 172.21.15.59:0/3587417207 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0444010b50 con 0x7f0454102be0 2024-03-19T03:06:29.849 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.105+0000 7f045a4ee700 1 -- 172.21.15.59:0/3587417207 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f045410fa30 con 0x7f0454102be0 2024-03-19T03:06:29.849 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.105+0000 7f0450ff9700 1 -- 172.21.15.59:0/3587417207 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f044401d050 con 0x7f0454102be0 2024-03-19T03:06:29.849 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.105+0000 7f0450ff9700 1 -- 172.21.15.59:0/3587417207 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0444022590 con 0x7f0454102be0 2024-03-19T03:06:29.849 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.105+0000 7f045a4ee700 1 -- 172.21.15.59:0/3587417207 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04541105a0 con 0x7f0454102be0 2024-03-19T03:06:29.849 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.106+0000 7f0450ff9700 1 -- 172.21.15.59:0/3587417207 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 2) v1 ==== 43124+0+0 (secure 0 0 0) 0x7f0444022790 con 0x7f0454102be0 2024-03-19T03:06:29.849 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.106+0000 7f045a4ee700 1 -- 172.21.15.59:0/3587417207 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f045405f5f0 con 0x7f0454102be0 2024-03-19T03:06:29.849 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.106+0000 7f0450ff9700 1 -- 172.21.15.59:0/3587417207 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0444014070 con 0x7f0454102be0 2024-03-19T03:06:29.849 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.108+0000 7f0450ff9700 1 -- 172.21.15.59:0/3587417207 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f0444023470 con 0x7f0454102be0 2024-03-19T03:06:29.850 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.181+0000 7f045a4ee700 1 -- 172.21.15.59:0/3587417207 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f045404bed0 con 0x7f0454102be0 2024-03-19T03:06:29.850 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.181+0000 7f0450ff9700 1 -- 172.21.15.59:0/3587417207 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f0444021b20 con 0x7f0454102be0 2024-03-19T03:06:29.850 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.181+0000 7f045a4ee700 1 -- 172.21.15.59:0/3587417207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454102be0 msgr2=0x7f045410f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:29.850 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.181+0000 7f045a4ee700 1 --2- 172.21.15.59:0/3587417207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454102be0 0x7f045410f2f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f044400d650 tx=0x7f044400b4f0 comp rx=0 tx=0).stop 2024-03-19T03:06:29.850 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.182+0000 7f045a4ee700 1 -- 172.21.15.59:0/3587417207 shutdown_connections 2024-03-19T03:06:29.850 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.182+0000 7f045a4ee700 1 --2- 172.21.15.59:0/3587417207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454102be0 0x7f045410f2f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:29.850 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.182+0000 7f045a4ee700 1 -- 172.21.15.59:0/3587417207 >> 172.21.15.59:0/3587417207 conn(0x7f04540fc500 msgr2=0x7f0454105b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:29.850 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.182+0000 7f045a4ee700 1 -- 172.21.15.59:0/3587417207 shutdown_connections 2024-03-19T03:06:29.850 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:29.182+0000 7f045a4ee700 1 -- 172.21.15.59:0/3587417207 wait complete. 2024-03-19T03:06:29.850 INFO:teuthology.orchestra.run.smithi059.stdout:mgr not available, waiting (3/15)... 2024-03-19T03:06:30.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:29 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/3587417207' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-19T03:06:30.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:29 smithi059 ceph-mon[29133]: mgrmap e3: smithi059.azpecc(active, since 1.00608s) 2024-03-19T03:06:32.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:32 smithi059 ceph-mon[29133]: mgrmap e4: smithi059.azpecc(active, since 2s) 2024-03-19T03:06:36.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:36 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/772266529' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-19T03:06:41.210 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-03-19T03:06:41.211 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2024-03-19T03:06:41.211 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "fsid": "8c8ae530-e59d-11ee-95c9-87774f69a715", 2024-03-19T03:06:41.211 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "health": { 2024-03-19T03:06:41.211 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-19T03:06:41.211 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-19T03:06:41.211 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-19T03:06:41.211 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:41.211 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-19T03:06:41.211 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-19T03:06:41.211 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 0 2024-03-19T03:06:41.211 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:06:41.211 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-19T03:06:41.211 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "smithi059" 2024-03-19T03:06:41.211 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:06:41.211 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-03-19T03:06:41.212 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-19T03:06:41.212 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:06:41.212 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-19T03:06:41.212 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-19T03:06:41.212 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:41.212 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-19T03:06:41.212 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:06:41.212 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-19T03:06:41.212 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-19T03:06:41.212 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-19T03:06:41.212 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-19T03:06:41.212 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-19T03:06:41.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-19T03:06:41.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:41.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-19T03:06:41.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-19T03:06:41.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-19T03:06:41.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-19T03:06:41.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-19T03:06:41.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-19T03:06:41.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-19T03:06:41.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-19T03:06:41.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-19T03:06:41.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:41.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-19T03:06:41.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:06:41.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-19T03:06:41.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-19T03:06:41.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:41.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-19T03:06:41.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-19T03:06:41.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-19T03:06:41.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-19T03:06:41.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-19T03:06:41.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-19T03:06:41.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "restful" 2024-03-19T03:06:41.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:06:41.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T03:06:41.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:41.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-19T03:06:41.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:06:41.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "modified": "2024-03-19T03:06:15.064565+0000", 2024-03-19T03:06:41.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T03:06:41.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:06:41.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-19T03:06:41.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2024-03-19T03:06:41.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.853+0000 7f91243fc700 1 Processor -- start 2024-03-19T03:06:41.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.853+0000 7f91243fc700 1 -- start start 2024-03-19T03:06:41.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.854+0000 7f91243fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f911c102720 0x7f911c102b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:41.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.854+0000 7f91243fc700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911c103070 con 0x7f911c102720 2024-03-19T03:06:41.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.854+0000 7f9122198700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f911c102720 0x7f911c102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:41.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.854+0000 7f9122198700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f911c102720 0x7f911c102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54852/0 (socket says 172.21.15.59:54852) 2024-03-19T03:06:41.224 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.854+0000 7f9122198700 1 -- 172.21.15.59:0/1353659240 learned_addr learned my addr 172.21.15.59:0/1353659240 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:41.224 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.854+0000 7f9122198700 1 -- 172.21.15.59:0/1353659240 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911c1031b0 con 0x7f911c102720 2024-03-19T03:06:41.224 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.854+0000 7f9122198700 1 --2- 172.21.15.59:0/1353659240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f911c102720 0x7f911c102b30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f91180098e0 tx=0x7f9118009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b826278ec148666b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:41.224 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.855+0000 7f9121196700 1 -- 172.21.15.59:0/1353659240 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f911800b860 con 0x7f911c102720 2024-03-19T03:06:41.224 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.855+0000 7f9121196700 1 -- 172.21.15.59:0/1353659240 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f911800be50 con 0x7f911c102720 2024-03-19T03:06:41.224 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.855+0000 7f9121196700 1 -- 172.21.15.59:0/1353659240 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9118010380 con 0x7f911c102720 2024-03-19T03:06:41.224 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.855+0000 7f91243fc700 1 -- 172.21.15.59:0/1353659240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f911c102720 msgr2=0x7f911c102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:41.224 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.855+0000 7f91243fc700 1 --2- 172.21.15.59:0/1353659240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f911c102720 0x7f911c102b30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f91180098e0 tx=0x7f9118009250 comp rx=0 tx=0).stop 2024-03-19T03:06:41.224 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.855+0000 7f91243fc700 1 -- 172.21.15.59:0/1353659240 shutdown_connections 2024-03-19T03:06:41.224 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.855+0000 7f91243fc700 1 --2- 172.21.15.59:0/1353659240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f911c102720 0x7f911c102b30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:41.224 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.855+0000 7f91243fc700 1 -- 172.21.15.59:0/1353659240 >> 172.21.15.59:0/1353659240 conn(0x7f911c0fdee0 msgr2=0x7f911c1002f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:41.224 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.855+0000 7f91243fc700 1 -- 172.21.15.59:0/1353659240 shutdown_connections 2024-03-19T03:06:41.224 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.855+0000 7f91243fc700 1 -- 172.21.15.59:0/1353659240 wait complete. 2024-03-19T03:06:41.224 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.856+0000 7f91243fc700 1 Processor -- start 2024-03-19T03:06:41.225 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.856+0000 7f91243fc700 1 -- start start 2024-03-19T03:06:41.225 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.856+0000 7f91243fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f911c102720 0x7f911c111560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:41.225 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.856+0000 7f91243fc700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911c111aa0 con 0x7f911c102720 2024-03-19T03:06:41.225 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.856+0000 7f9122198700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f911c102720 0x7f911c111560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:41.225 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.856+0000 7f9122198700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f911c102720 0x7f911c111560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54854/0 (socket says 172.21.15.59:54854) 2024-03-19T03:06:41.225 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.856+0000 7f9122198700 1 -- 172.21.15.59:0/772266529 learned_addr learned my addr 172.21.15.59:0/772266529 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:41.225 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.857+0000 7f9122198700 1 -- 172.21.15.59:0/772266529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91180094f0 con 0x7f911c102720 2024-03-19T03:06:41.225 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.857+0000 7f9122198700 1 --2- 172.21.15.59:0/772266529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f911c102720 0x7f911c111560 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f911800b440 tx=0x7f9118009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:41.225 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.857+0000 7f910effd700 1 -- 172.21.15.59:0/772266529 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9118010760 con 0x7f911c102720 2024-03-19T03:06:41.225 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.857+0000 7f910effd700 1 -- 172.21.15.59:0/772266529 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f9118010d50 con 0x7f911c102720 2024-03-19T03:06:41.225 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.857+0000 7f91243fc700 1 -- 172.21.15.59:0/772266529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f911c111ca0 con 0x7f911c102720 2024-03-19T03:06:41.225 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.857+0000 7f910effd700 1 -- 172.21.15.59:0/772266529 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f911801ae20 con 0x7f911c102720 2024-03-19T03:06:41.225 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.857+0000 7f91243fc700 1 -- 172.21.15.59:0/772266529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f911c112810 con 0x7f911c102720 2024-03-19T03:06:41.225 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.858+0000 7f910effd700 1 -- 172.21.15.59:0/772266529 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f91180108c0 con 0x7f911c102720 2024-03-19T03:06:41.226 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.858+0000 7f91243fc700 1 -- 172.21.15.59:0/772266529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f911c05f5f0 con 0x7f911c102720 2024-03-19T03:06:41.226 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.858+0000 7f910effd700 1 --2- 172.21.15.59:0/772266529 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f9108036790 0x7f9108038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:41.226 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.858+0000 7f910effd700 1 -- 172.21.15.59:0/772266529 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f911804a4c0 con 0x7f911c102720 2024-03-19T03:06:41.226 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.858+0000 7f9121997700 1 --2- 172.21.15.59:0/772266529 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f9108036790 0x7f9108038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:41.226 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.859+0000 7f9121997700 1 --2- 172.21.15.59:0/772266529 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f9108036790 0x7f9108038c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9110009990 tx=0x7f9110006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:41.226 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:34.863+0000 7f910effd700 1 -- 172.21.15.59:0/772266529 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f91180187c0 con 0x7f911c102720 2024-03-19T03:06:41.226 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:35.135+0000 7f91243fc700 1 -- 172.21.15.59:0/772266529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f911c04bed0 con 0x7f911c102720 2024-03-19T03:06:41.226 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:35.135+0000 7f910effd700 1 -- 172.21.15.59:0/772266529 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f91180187c0 con 0x7f911c102720 2024-03-19T03:06:41.226 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:35.137+0000 7f91243fc700 1 -- 172.21.15.59:0/772266529 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f9108036790 msgr2=0x7f9108038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:41.226 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:35.137+0000 7f91243fc700 1 --2- 172.21.15.59:0/772266529 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f9108036790 0x7f9108038c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9110009990 tx=0x7f9110006e40 comp rx=0 tx=0).stop 2024-03-19T03:06:41.226 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:35.137+0000 7f91243fc700 1 -- 172.21.15.59:0/772266529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f911c102720 msgr2=0x7f911c111560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:41.226 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:35.137+0000 7f91243fc700 1 --2- 172.21.15.59:0/772266529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f911c102720 0x7f911c111560 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f911800b440 tx=0x7f9118009250 comp rx=0 tx=0).stop 2024-03-19T03:06:41.226 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:35.138+0000 7f91243fc700 1 -- 172.21.15.59:0/772266529 shutdown_connections 2024-03-19T03:06:41.227 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:35.138+0000 7f91243fc700 1 --2- 172.21.15.59:0/772266529 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f9108036790 0x7f9108038c40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:41.227 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:35.138+0000 7f91243fc700 1 --2- 172.21.15.59:0/772266529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f911c102720 0x7f911c111560 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:41.227 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:35.138+0000 7f91243fc700 1 -- 172.21.15.59:0/772266529 >> 172.21.15.59:0/772266529 conn(0x7f911c0fdee0 msgr2=0x7f911c0fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:41.227 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:35.138+0000 7f91243fc700 1 -- 172.21.15.59:0/772266529 shutdown_connections 2024-03-19T03:06:41.227 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:35.138+0000 7f91243fc700 1 -- 172.21.15.59:0/772266529 wait complete. 2024-03-19T03:06:41.227 INFO:teuthology.orchestra.run.smithi059.stdout:mgr is available 2024-03-19T03:06:42.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:41 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/3114264233' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-19T03:06:42.497 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-03-19T03:06:42.504 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout [global] 2024-03-19T03:06:42.504 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout fsid = 8c8ae530-e59d-11ee-95c9-87774f69a715 2024-03-19T03:06:42.504 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-19T03:06:42.504 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-19T03:06:42.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-19T03:06:42.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-19T03:06:42.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-03-19T03:06:42.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-19T03:06:42.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-19T03:06:42.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-03-19T03:06:42.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout [osd] 2024-03-19T03:06:42.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-19T03:06:42.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-19T03:06:42.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-19T03:06:42.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-19T03:06:42.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-19T03:06:42.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.627+0000 7f6685cb2700 1 Processor -- start 2024-03-19T03:06:42.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.627+0000 7f6685cb2700 1 -- start start 2024-03-19T03:06:42.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.627+0000 7f6685cb2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6680101c90 0x7f66801020a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:42.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.627+0000 7f6685cb2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66801025e0 con 0x7f6680101c90 2024-03-19T03:06:42.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.628+0000 7f667f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6680101c90 0x7f66801020a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:42.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.628+0000 7f667f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6680101c90 0x7f66801020a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54858/0 (socket says 172.21.15.59:54858) 2024-03-19T03:06:42.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.628+0000 7f667f7fe700 1 -- 172.21.15.59:0/1208706751 learned_addr learned my addr 172.21.15.59:0/1208706751 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:42.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.628+0000 7f667f7fe700 1 -- 172.21.15.59:0/1208706751 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6680102720 con 0x7f6680101c90 2024-03-19T03:06:42.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.628+0000 7f667f7fe700 1 --2- 172.21.15.59:0/1208706751 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6680101c90 0x7f66801020a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6678009e50 tx=0x7f6678009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8408bd882385b44a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:42.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.628+0000 7f667e7fc700 1 -- 172.21.15.59:0/1208706751 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66780092e0 con 0x7f6680101c90 2024-03-19T03:06:42.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.628+0000 7f667e7fc700 1 -- 172.21.15.59:0/1208706751 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6678012430 con 0x7f6680101c90 2024-03-19T03:06:42.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.628+0000 7f667e7fc700 1 -- 172.21.15.59:0/1208706751 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66780075c0 con 0x7f6680101c90 2024-03-19T03:06:42.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.629+0000 7f6685cb2700 1 -- 172.21.15.59:0/1208706751 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6680101c90 msgr2=0x7f66801020a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:42.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.629+0000 7f6685cb2700 1 --2- 172.21.15.59:0/1208706751 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6680101c90 0x7f66801020a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6678009e50 tx=0x7f6678009840 comp rx=0 tx=0).stop 2024-03-19T03:06:42.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.629+0000 7f6685cb2700 1 -- 172.21.15.59:0/1208706751 shutdown_connections 2024-03-19T03:06:42.507 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.629+0000 7f6685cb2700 1 --2- 172.21.15.59:0/1208706751 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6680101c90 0x7f66801020a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:42.507 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.629+0000 7f6685cb2700 1 -- 172.21.15.59:0/1208706751 >> 172.21.15.59:0/1208706751 conn(0x7f66800fc560 msgr2=0x7f66800fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:42.507 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.629+0000 7f6685cb2700 1 -- 172.21.15.59:0/1208706751 shutdown_connections 2024-03-19T03:06:42.507 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.629+0000 7f6685cb2700 1 -- 172.21.15.59:0/1208706751 wait complete. 2024-03-19T03:06:42.507 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.629+0000 7f6685cb2700 1 Processor -- start 2024-03-19T03:06:42.507 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.630+0000 7f6685cb2700 1 -- start start 2024-03-19T03:06:42.507 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.630+0000 7f6685cb2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6680101c90 0x7f6680101490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:42.507 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.630+0000 7f6685cb2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6680114340 con 0x7f6680101c90 2024-03-19T03:06:42.507 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.630+0000 7f667f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6680101c90 0x7f6680101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:42.507 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.630+0000 7f667f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6680101c90 0x7f6680101490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54874/0 (socket says 172.21.15.59:54874) 2024-03-19T03:06:42.507 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.630+0000 7f667f7fe700 1 -- 172.21.15.59:0/3114264233 learned_addr learned my addr 172.21.15.59:0/3114264233 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:42.507 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.630+0000 7f667f7fe700 1 -- 172.21.15.59:0/3114264233 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6678009ae0 con 0x7f6680101c90 2024-03-19T03:06:42.507 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.630+0000 7f667f7fe700 1 --2- 172.21.15.59:0/3114264233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6680101c90 0x7f6680101490 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6678012da0 tx=0x7f6678009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:42.507 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.630+0000 7f6677fff700 1 -- 172.21.15.59:0/3114264233 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66780092e0 con 0x7f6680101c90 2024-03-19T03:06:42.508 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.630+0000 7f6677fff700 1 -- 172.21.15.59:0/3114264233 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6678007cc0 con 0x7f6680101c90 2024-03-19T03:06:42.508 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.630+0000 7f6685cb2700 1 -- 172.21.15.59:0/3114264233 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6680114480 con 0x7f6680101c90 2024-03-19T03:06:42.508 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.631+0000 7f6677fff700 1 -- 172.21.15.59:0/3114264233 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f667801c910 con 0x7f6680101c90 2024-03-19T03:06:42.508 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.631+0000 7f6685cb2700 1 -- 172.21.15.59:0/3114264233 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6680114970 con 0x7f6680101c90 2024-03-19T03:06:42.508 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.631+0000 7f6677fff700 1 -- 172.21.15.59:0/3114264233 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f6678019070 con 0x7f6680101c90 2024-03-19T03:06:42.508 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.631+0000 7f6677fff700 1 --2- 172.21.15.59:0/3114264233 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f666c036740 0x7f666c038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:42.508 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.632+0000 7f6677fff700 1 -- 172.21.15.59:0/3114264233 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f6678049ee0 con 0x7f6680101c90 2024-03-19T03:06:42.508 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.632+0000 7f6685cb2700 1 -- 172.21.15.59:0/3114264233 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f668004bed0 con 0x7f6680101c90 2024-03-19T03:06:42.508 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.632+0000 7f667effd700 1 --2- 172.21.15.59:0/3114264233 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f666c036740 0x7f666c038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:42.508 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.632+0000 7f667effd700 1 --2- 172.21.15.59:0/3114264233 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f666c036740 0x7f666c038bf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6668009990 tx=0x7f6668006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:42.508 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.636+0000 7f6677fff700 1 -- 172.21.15.59:0/3114264233 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f6678075e50 con 0x7f6680101c90 2024-03-19T03:06:42.509 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.827+0000 7f6685cb2700 1 -- 172.21.15.59:0/3114264233 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f66801034d0 con 0x7f6680101c90 2024-03-19T03:06:42.509 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.828+0000 7f6677fff700 1 -- 172.21.15.59:0/3114264233 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f6678034020 con 0x7f6680101c90 2024-03-19T03:06:42.509 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.830+0000 7f6685cb2700 1 -- 172.21.15.59:0/3114264233 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f666c036740 msgr2=0x7f666c038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:42.509 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.830+0000 7f6685cb2700 1 --2- 172.21.15.59:0/3114264233 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f666c036740 0x7f666c038bf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6668009990 tx=0x7f6668006e40 comp rx=0 tx=0).stop 2024-03-19T03:06:42.509 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.830+0000 7f6685cb2700 1 -- 172.21.15.59:0/3114264233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6680101c90 msgr2=0x7f6680101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:42.509 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.830+0000 7f6685cb2700 1 --2- 172.21.15.59:0/3114264233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6680101c90 0x7f6680101490 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6678012da0 tx=0x7f6678009840 comp rx=0 tx=0).stop 2024-03-19T03:06:42.509 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.830+0000 7f6685cb2700 1 -- 172.21.15.59:0/3114264233 shutdown_connections 2024-03-19T03:06:42.509 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.830+0000 7f6685cb2700 1 --2- 172.21.15.59:0/3114264233 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f666c036740 0x7f666c038bf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:42.509 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.830+0000 7f6685cb2700 1 --2- 172.21.15.59:0/3114264233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6680101c90 0x7f6680101490 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:42.509 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.830+0000 7f6685cb2700 1 -- 172.21.15.59:0/3114264233 >> 172.21.15.59:0/3114264233 conn(0x7f66800fc560 msgr2=0x7f66800fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:42.509 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.831+0000 7f6685cb2700 1 -- 172.21.15.59:0/3114264233 shutdown_connections 2024-03-19T03:06:42.509 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:41.831+0000 7f6685cb2700 1 -- 172.21.15.59:0/3114264233 wait complete. 2024-03-19T03:06:42.509 INFO:teuthology.orchestra.run.smithi059.stdout:Enabling cephadm module... 2024-03-19T03:06:43.458 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:43 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/3336515330' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-19T03:06:43.730 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.849+0000 7f731aadb700 1 Processor -- start 2024-03-19T03:06:43.730 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.849+0000 7f731aadb700 1 -- start start 2024-03-19T03:06:43.730 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.849+0000 7f731aadb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7314101c00 0x7f7314102010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:43.730 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.849+0000 7f731aadb700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7314102550 con 0x7f7314101c00 2024-03-19T03:06:43.730 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.849+0000 7f7318877700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7314101c00 0x7f7314102010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:43.730 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.849+0000 7f7318877700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7314101c00 0x7f7314102010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54886/0 (socket says 172.21.15.59:54886) 2024-03-19T03:06:43.731 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.849+0000 7f7318877700 1 -- 172.21.15.59:0/2216532544 learned_addr learned my addr 172.21.15.59:0/2216532544 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:43.731 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.849+0000 7f7318877700 1 -- 172.21.15.59:0/2216532544 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7314102690 con 0x7f7314101c00 2024-03-19T03:06:43.731 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.850+0000 7f7318877700 1 --2- 172.21.15.59:0/2216532544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7314101c00 0x7f7314102010 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f730800bb80 tx=0x7f730800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=476885af20a0816e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:43.731 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.850+0000 7f73137fe700 1 -- 172.21.15.59:0/2216532544 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f730800dd00 con 0x7f7314101c00 2024-03-19T03:06:43.731 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.850+0000 7f73137fe700 1 -- 172.21.15.59:0/2216532544 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f730800de60 con 0x7f7314101c00 2024-03-19T03:06:43.731 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.850+0000 7f73137fe700 1 -- 172.21.15.59:0/2216532544 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73080188a0 con 0x7f7314101c00 2024-03-19T03:06:43.731 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.850+0000 7f731aadb700 1 -- 172.21.15.59:0/2216532544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7314101c00 msgr2=0x7f7314102010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:43.731 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.850+0000 7f731aadb700 1 --2- 172.21.15.59:0/2216532544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7314101c00 0x7f7314102010 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f730800bb80 tx=0x7f730800b4f0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.731 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.850+0000 7f731aadb700 1 -- 172.21.15.59:0/2216532544 shutdown_connections 2024-03-19T03:06:43.731 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.850+0000 7f731aadb700 1 --2- 172.21.15.59:0/2216532544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7314101c00 0x7f7314102010 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.731 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.850+0000 7f731aadb700 1 -- 172.21.15.59:0/2216532544 >> 172.21.15.59:0/2216532544 conn(0x7f73140fc4d0 msgr2=0x7f73140fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:43.731 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.850+0000 7f731aadb700 1 -- 172.21.15.59:0/2216532544 shutdown_connections 2024-03-19T03:06:43.731 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.850+0000 7f731aadb700 1 -- 172.21.15.59:0/2216532544 wait complete. 2024-03-19T03:06:43.732 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.851+0000 7f731aadb700 1 Processor -- start 2024-03-19T03:06:43.732 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.851+0000 7f731aadb700 1 -- start start 2024-03-19T03:06:43.732 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.851+0000 7f731aadb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7314101c00 0x7f731410f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:43.732 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.851+0000 7f731aadb700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731410f820 con 0x7f7314101c00 2024-03-19T03:06:43.732 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.851+0000 7f7318877700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7314101c00 0x7f731410f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:43.732 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.851+0000 7f7318877700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7314101c00 0x7f731410f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54896/0 (socket says 172.21.15.59:54896) 2024-03-19T03:06:43.732 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.851+0000 7f7318877700 1 -- 172.21.15.59:0/3336515330 learned_addr learned my addr 172.21.15.59:0/3336515330 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:43.732 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.852+0000 7f7318877700 1 -- 172.21.15.59:0/3336515330 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730800b790 con 0x7f7314101c00 2024-03-19T03:06:43.732 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.852+0000 7f7318877700 1 --2- 172.21.15.59:0/3336515330 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7314101c00 0x7f731410f2e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f730800d650 tx=0x7f730800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:43.732 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.852+0000 7f73117fa700 1 -- 172.21.15.59:0/3336515330 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7308010b50 con 0x7f7314101c00 2024-03-19T03:06:43.732 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.852+0000 7f73117fa700 1 -- 172.21.15.59:0/3336515330 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f730801d050 con 0x7f7314101c00 2024-03-19T03:06:43.732 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.852+0000 7f731aadb700 1 -- 172.21.15.59:0/3336515330 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731410fa20 con 0x7f7314101c00 2024-03-19T03:06:43.732 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.852+0000 7f731aadb700 1 -- 172.21.15.59:0/3336515330 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7314110590 con 0x7f7314101c00 2024-03-19T03:06:43.733 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.852+0000 7f73117fa700 1 -- 172.21.15.59:0/3336515330 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7308022590 con 0x7f7314101c00 2024-03-19T03:06:43.733 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.853+0000 7f73117fa700 1 -- 172.21.15.59:0/3336515330 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f73080227b0 con 0x7f7314101c00 2024-03-19T03:06:43.733 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.853+0000 7f73117fa700 1 --2- 172.21.15.59:0/3336515330 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f7304036720 0x7f7304038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:43.733 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.853+0000 7f73117fa700 1 -- 172.21.15.59:0/3336515330 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f7308014070 con 0x7f7314101c00 2024-03-19T03:06:43.733 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.853+0000 7f731aadb700 1 -- 172.21.15.59:0/3336515330 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f731405f5f0 con 0x7f7314101c00 2024-03-19T03:06:43.733 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.853+0000 7f7313fff700 1 --2- 172.21.15.59:0/3336515330 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f7304036720 0x7f7304038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:43.733 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.854+0000 7f7313fff700 1 --2- 172.21.15.59:0/3336515330 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f7304036720 0x7f7304038bd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7300009990 tx=0x7f7300006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:43.733 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:42.859+0000 7f73117fa700 1 -- 172.21.15.59:0/3336515330 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f7308021b90 con 0x7f7314101c00 2024-03-19T03:06:43.733 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:43.103+0000 7f731aadb700 1 -- 172.21.15.59:0/3336515330 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f7314110fd0 con 0x7f7314101c00 2024-03-19T03:06:43.733 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:43.155+0000 7f73117fa700 1 -- 172.21.15.59:0/3336515330 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f7308021dc0 con 0x7f7314101c00 2024-03-19T03:06:43.733 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:43.156+0000 7f73117fa700 1 -- 172.21.15.59:0/3336515330 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f73080217a0 con 0x7f7314101c00 2024-03-19T03:06:43.733 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:43.159+0000 7f731aadb700 1 -- 172.21.15.59:0/3336515330 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f7304036720 msgr2=0x7f7304038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:43.733 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:43.159+0000 7f731aadb700 1 --2- 172.21.15.59:0/3336515330 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f7304036720 0x7f7304038bd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7300009990 tx=0x7f7300006e40 comp rx=0 tx=0).stop 2024-03-19T03:06:43.733 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:43.159+0000 7f731aadb700 1 -- 172.21.15.59:0/3336515330 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7314101c00 msgr2=0x7f731410f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:43.734 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:43.159+0000 7f731aadb700 1 --2- 172.21.15.59:0/3336515330 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7314101c00 0x7f731410f2e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f730800d650 tx=0x7f730800b4f0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.734 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:43.159+0000 7f731aadb700 1 -- 172.21.15.59:0/3336515330 shutdown_connections 2024-03-19T03:06:43.734 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:43.159+0000 7f731aadb700 1 --2- 172.21.15.59:0/3336515330 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f7304036720 0x7f7304038bd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.734 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:43.159+0000 7f731aadb700 1 --2- 172.21.15.59:0/3336515330 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7314101c00 0x7f731410f2e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:43.734 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:43.159+0000 7f731aadb700 1 -- 172.21.15.59:0/3336515330 >> 172.21.15.59:0/3336515330 conn(0x7f73140fc4d0 msgr2=0x7f7314105bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:43.734 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:43.159+0000 7f731aadb700 1 -- 172.21.15.59:0/3336515330 shutdown_connections 2024-03-19T03:06:43.734 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:43.159+0000 7f731aadb700 1 -- 172.21.15.59:0/3336515330 wait complete. 2024-03-19T03:06:44.330 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:44 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/3336515330' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-19T03:06:44.330 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:44 smithi059 ceph-mon[29133]: mgrmap e5: smithi059.azpecc(active, since 14s) 2024-03-19T03:06:44.891 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2024-03-19T03:06:44.891 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-19T03:06:44.891 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-19T03:06:44.891 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "active_name": "smithi059.azpecc", 2024-03-19T03:06:44.891 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-19T03:06:44.891 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2024-03-19T03:06:44.891 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.123+0000 7f135270a700 1 Processor -- start 2024-03-19T03:06:44.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.123+0000 7f135270a700 1 -- start start 2024-03-19T03:06:44.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.123+0000 7f135270a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f134c076b60 0x7f134c105370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:44.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.123+0000 7f135270a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c1058b0 con 0x7f134c076b60 2024-03-19T03:06:44.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.123+0000 7f134bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f134c076b60 0x7f134c105370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:44.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.123+0000 7f134bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f134c076b60 0x7f134c105370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48766/0 (socket says 172.21.15.59:48766) 2024-03-19T03:06:44.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.123+0000 7f134bfff700 1 -- 172.21.15.59:0/3498089852 learned_addr learned my addr 172.21.15.59:0/3498089852 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:44.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.123+0000 7f134bfff700 1 -- 172.21.15.59:0/3498089852 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134c1059f0 con 0x7f134c076b60 2024-03-19T03:06:44.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.124+0000 7f134bfff700 1 --2- 172.21.15.59:0/3498089852 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f134c076b60 0x7f134c105370 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f133c0098e0 tx=0x7f133c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe96282c6e34b18 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:44.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.124+0000 7f134affd700 1 -- 172.21.15.59:0/3498089852 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f133c00b860 con 0x7f134c076b60 2024-03-19T03:06:44.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.124+0000 7f134affd700 1 -- 172.21.15.59:0/3498089852 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f133c00be50 con 0x7f134c076b60 2024-03-19T03:06:44.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.124+0000 7f134affd700 1 -- 172.21.15.59:0/3498089852 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f133c010380 con 0x7f134c076b60 2024-03-19T03:06:44.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.124+0000 7f135270a700 1 -- 172.21.15.59:0/3498089852 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f134c076b60 msgr2=0x7f134c105370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:44.893 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.124+0000 7f135270a700 1 --2- 172.21.15.59:0/3498089852 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f134c076b60 0x7f134c105370 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f133c0098e0 tx=0x7f133c009250 comp rx=0 tx=0).stop 2024-03-19T03:06:44.893 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.124+0000 7f135270a700 1 -- 172.21.15.59:0/3498089852 shutdown_connections 2024-03-19T03:06:44.893 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.124+0000 7f135270a700 1 --2- 172.21.15.59:0/3498089852 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f134c076b60 0x7f134c105370 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.893 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.124+0000 7f135270a700 1 -- 172.21.15.59:0/3498089852 >> 172.21.15.59:0/3498089852 conn(0x7f134c0fcd80 msgr2=0x7f134c0ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:44.893 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.124+0000 7f135270a700 1 -- 172.21.15.59:0/3498089852 shutdown_connections 2024-03-19T03:06:44.893 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.124+0000 7f135270a700 1 -- 172.21.15.59:0/3498089852 wait complete. 2024-03-19T03:06:44.893 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.125+0000 7f135270a700 1 Processor -- start 2024-03-19T03:06:44.893 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.125+0000 7f135270a700 1 -- start start 2024-03-19T03:06:44.893 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.125+0000 7f135270a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f134c076b60 0x7f134c102cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:44.893 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.125+0000 7f135270a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c103200 con 0x7f134c076b60 2024-03-19T03:06:44.893 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.125+0000 7f134bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f134c076b60 0x7f134c102cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:44.893 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.125+0000 7f134bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f134c076b60 0x7f134c102cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48780/0 (socket says 172.21.15.59:48780) 2024-03-19T03:06:44.894 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.125+0000 7f134bfff700 1 -- 172.21.15.59:0/807405653 learned_addr learned my addr 172.21.15.59:0/807405653 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:44.894 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.126+0000 7f134bfff700 1 -- 172.21.15.59:0/807405653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133c0094f0 con 0x7f134c076b60 2024-03-19T03:06:44.894 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.126+0000 7f134bfff700 1 --2- 172.21.15.59:0/807405653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f134c076b60 0x7f134c102cc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f133c003050 tx=0x7f133c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:44.894 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.126+0000 7f1348ff9700 1 -- 172.21.15.59:0/807405653 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f133c010760 con 0x7f134c076b60 2024-03-19T03:06:44.894 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.126+0000 7f1348ff9700 1 -- 172.21.15.59:0/807405653 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f133c010d50 con 0x7f134c076b60 2024-03-19T03:06:44.894 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.126+0000 7f1348ff9700 1 -- 172.21.15.59:0/807405653 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f133c01ae20 con 0x7f134c076b60 2024-03-19T03:06:44.894 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.126+0000 7f135270a700 1 -- 172.21.15.59:0/807405653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134c101270 con 0x7f134c076b60 2024-03-19T03:06:44.894 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.126+0000 7f135270a700 1 -- 172.21.15.59:0/807405653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f134c101760 con 0x7f134c076b60 2024-03-19T03:06:44.894 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.127+0000 7f1348ff9700 1 -- 172.21.15.59:0/807405653 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f133c02e440 con 0x7f134c076b60 2024-03-19T03:06:44.895 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.127+0000 7f135270a700 1 -- 172.21.15.59:0/807405653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f134c04bed0 con 0x7f134c076b60 2024-03-19T03:06:44.895 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.127+0000 7f1348ff9700 1 --2- 172.21.15.59:0/807405653 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f13340367e0 0x7f1334038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:44.895 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.127+0000 7f1348ff9700 1 -- 172.21.15.59:0/807405653 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f133c04a4a0 con 0x7f134c076b60 2024-03-19T03:06:44.895 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.127+0000 7f134b7fe700 1 -- 172.21.15.59:0/807405653 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f13340367e0 msgr2=0x7f1334038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3609017287 2024-03-19T03:06:44.895 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.127+0000 7f134b7fe700 1 --2- 172.21.15.59:0/807405653 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f13340367e0 0x7f1334038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T03:06:44.895 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.132+0000 7f1348ff9700 1 -- 172.21.15.59:0/807405653 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f133c018940 con 0x7f134c076b60 2024-03-19T03:06:44.895 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.328+0000 7f134b7fe700 1 -- 172.21.15.59:0/807405653 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f13340367e0 msgr2=0x7f1334038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3609017287 2024-03-19T03:06:44.895 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.328+0000 7f134b7fe700 1 --2- 172.21.15.59:0/807405653 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f13340367e0 0x7f1334038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T03:06:44.895 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.378+0000 7f135270a700 1 -- 172.21.15.59:0/807405653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f134c101e00 con 0x7f134c076b60 2024-03-19T03:06:44.895 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.378+0000 7f1348ff9700 1 -- 172.21.15.59:0/807405653 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f133c02d3a0 con 0x7f134c076b60 2024-03-19T03:06:44.895 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.381+0000 7f135270a700 1 -- 172.21.15.59:0/807405653 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f13340367e0 msgr2=0x7f1334038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T03:06:44.896 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.381+0000 7f135270a700 1 --2- 172.21.15.59:0/807405653 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f13340367e0 0x7f1334038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.896 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.381+0000 7f135270a700 1 -- 172.21.15.59:0/807405653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f134c076b60 msgr2=0x7f134c102cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:44.896 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.381+0000 7f135270a700 1 --2- 172.21.15.59:0/807405653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f134c076b60 0x7f134c102cc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f133c003050 tx=0x7f133c009250 comp rx=0 tx=0).stop 2024-03-19T03:06:44.896 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.381+0000 7f135270a700 1 -- 172.21.15.59:0/807405653 shutdown_connections 2024-03-19T03:06:44.896 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.381+0000 7f135270a700 1 --2- 172.21.15.59:0/807405653 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f13340367e0 0x7f1334038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.896 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.381+0000 7f135270a700 1 --2- 172.21.15.59:0/807405653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f134c076b60 0x7f134c102cc0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:44.896 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.381+0000 7f135270a700 1 -- 172.21.15.59:0/807405653 >> 172.21.15.59:0/807405653 conn(0x7f134c0fcd80 msgr2=0x7f134c0fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:44.896 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.381+0000 7f135270a700 1 -- 172.21.15.59:0/807405653 shutdown_connections 2024-03-19T03:06:44.896 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:44.381+0000 7f135270a700 1 -- 172.21.15.59:0/807405653 wait complete. 2024-03-19T03:06:44.896 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for the mgr to restart... 2024-03-19T03:06:44.896 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for mgr epoch 5... 2024-03-19T03:06:45.484 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:45 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/807405653' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-19T03:06:50.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:49 smithi059 ceph-mon[29133]: Active manager daemon smithi059.azpecc restarted 2024-03-19T03:06:50.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:49 smithi059 ceph-mon[29133]: Activating manager daemon smithi059.azpecc 2024-03-19T03:06:50.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:49 smithi059 ceph-mon[29133]: osdmap e2: 0 total, 0 up, 0 in 2024-03-19T03:06:50.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:49 smithi059 ceph-mon[29133]: mgrmap e6: smithi059.azpecc(active, starting, since 0.00385579s) 2024-03-19T03:06:50.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:49 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-03-19T03:06:50.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:49 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mgr metadata", "who": "smithi059.azpecc", "id": "smithi059.azpecc"}]: dispatch 2024-03-19T03:06:50.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:49 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T03:06:50.341 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:49 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T03:06:50.341 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:49 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T03:06:50.341 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:49 smithi059 ceph-mon[29133]: Manager daemon smithi059.azpecc is now available 2024-03-19T03:06:50.341 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:49 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' 2024-03-19T03:06:50.341 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:49 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' 2024-03-19T03:06:50.341 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:49 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:06:50.341 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:49 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:06:50.341 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:49 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:06:51.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:50 smithi059 ceph-mon[29133]: Found migration_current of "None". Setting to last migration. 2024-03-19T03:06:51.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:50 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.azpecc/mirror_snapshot_schedule"}]: dispatch 2024-03-19T03:06:51.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:50 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.azpecc/trash_purge_schedule"}]: dispatch 2024-03-19T03:06:51.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:50 smithi059 ceph-mon[29133]: mgrmap e7: smithi059.azpecc(active, since 1.00521s) 2024-03-19T03:06:51.331 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2024-03-19T03:06:51.331 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-19T03:06:51.331 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-19T03:06:51.331 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2024-03-19T03:06:51.331 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.314+0000 7f087d26f700 1 Processor -- start 2024-03-19T03:06:51.331 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.314+0000 7f087d26f700 1 -- start start 2024-03-19T03:06:51.331 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.314+0000 7f087d26f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0878076c30 0x7f08781052c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:51.332 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.314+0000 7f087d26f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0878105800 con 0x7f0878076c30 2024-03-19T03:06:51.332 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.314+0000 7f0876ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0878076c30 0x7f08781052c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:51.332 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.314+0000 7f0876ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0878076c30 0x7f08781052c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48782/0 (socket says 172.21.15.59:48782) 2024-03-19T03:06:51.332 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.314+0000 7f0876ffd700 1 -- 172.21.15.59:0/2334830663 learned_addr learned my addr 172.21.15.59:0/2334830663 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:51.332 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.315+0000 7f0876ffd700 1 -- 172.21.15.59:0/2334830663 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0878105940 con 0x7f0878076c30 2024-03-19T03:06:51.332 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.315+0000 7f0876ffd700 1 --2- 172.21.15.59:0/2334830663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0878076c30 0x7f08781052c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f086c0098e0 tx=0x7f086c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f6e454be3584aa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:51.332 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.315+0000 7f0875ffb700 1 -- 172.21.15.59:0/2334830663 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f086c00b860 con 0x7f0878076c30 2024-03-19T03:06:51.332 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.315+0000 7f0875ffb700 1 -- 172.21.15.59:0/2334830663 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f086c00be50 con 0x7f0878076c30 2024-03-19T03:06:51.332 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.315+0000 7f0875ffb700 1 -- 172.21.15.59:0/2334830663 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f086c010380 con 0x7f0878076c30 2024-03-19T03:06:51.332 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.316+0000 7f087d26f700 1 -- 172.21.15.59:0/2334830663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0878076c30 msgr2=0x7f08781052c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:51.332 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.316+0000 7f087d26f700 1 --2- 172.21.15.59:0/2334830663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0878076c30 0x7f08781052c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f086c0098e0 tx=0x7f086c009250 comp rx=0 tx=0).stop 2024-03-19T03:06:51.332 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.316+0000 7f087d26f700 1 -- 172.21.15.59:0/2334830663 shutdown_connections 2024-03-19T03:06:51.332 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.316+0000 7f087d26f700 1 --2- 172.21.15.59:0/2334830663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0878076c30 0x7f08781052c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:51.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.316+0000 7f087d26f700 1 -- 172.21.15.59:0/2334830663 >> 172.21.15.59:0/2334830663 conn(0x7f08780fcd50 msgr2=0x7f08780ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:51.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.316+0000 7f087d26f700 1 -- 172.21.15.59:0/2334830663 shutdown_connections 2024-03-19T03:06:51.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.316+0000 7f087d26f700 1 -- 172.21.15.59:0/2334830663 wait complete. 2024-03-19T03:06:51.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.316+0000 7f087d26f700 1 Processor -- start 2024-03-19T03:06:51.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.317+0000 7f087d26f700 1 -- start start 2024-03-19T03:06:51.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.317+0000 7f087d26f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0878076c30 0x7f0878102c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:51.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.317+0000 7f087d26f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0878103150 con 0x7f0878076c30 2024-03-19T03:06:51.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.317+0000 7f0876ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0878076c30 0x7f0878102c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:51.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.317+0000 7f0876ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0878076c30 0x7f0878102c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48796/0 (socket says 172.21.15.59:48796) 2024-03-19T03:06:51.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.317+0000 7f0876ffd700 1 -- 172.21.15.59:0/2405486810 learned_addr learned my addr 172.21.15.59:0/2405486810 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:51.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.317+0000 7f0876ffd700 1 -- 172.21.15.59:0/2405486810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c0094f0 con 0x7f0878076c30 2024-03-19T03:06:51.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.317+0000 7f0876ffd700 1 --2- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0878076c30 0x7f0878102c10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f086c003050 tx=0x7f086c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:51.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.317+0000 7f085f7fe700 1 -- 172.21.15.59:0/2405486810 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f086c010760 con 0x7f0878076c30 2024-03-19T03:06:51.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.317+0000 7f085f7fe700 1 -- 172.21.15.59:0/2405486810 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f086c010d50 con 0x7f0878076c30 2024-03-19T03:06:51.333 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.317+0000 7f085f7fe700 1 -- 172.21.15.59:0/2405486810 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f086c01ae20 con 0x7f0878076c30 2024-03-19T03:06:51.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.317+0000 7f087d26f700 1 -- 172.21.15.59:0/2405486810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08781011c0 con 0x7f0878076c30 2024-03-19T03:06:51.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.317+0000 7f087d26f700 1 -- 172.21.15.59:0/2405486810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08781016b0 con 0x7f0878076c30 2024-03-19T03:06:51.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.318+0000 7f085f7fe700 1 -- 172.21.15.59:0/2405486810 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f086c02e440 con 0x7f0878076c30 2024-03-19T03:06:51.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.319+0000 7f085f7fe700 1 --2- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f08600367e0 0x7f0860038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:51.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.319+0000 7f087d26f700 1 -- 172.21.15.59:0/2405486810 --> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f087804bed0 con 0x7f08600367e0 2024-03-19T03:06:51.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.319+0000 7f085f7fe700 1 -- 172.21.15.59:0/2405486810 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f086c04a530 con 0x7f0878076c30 2024-03-19T03:06:51.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.319+0000 7f08767fc700 1 -- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f08600367e0 msgr2=0x7f0860038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3609017287 2024-03-19T03:06:51.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.319+0000 7f08767fc700 1 --2- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f08600367e0 0x7f0860038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T03:06:51.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.520+0000 7f08767fc700 1 -- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f08600367e0 msgr2=0x7f0860038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3609017287 2024-03-19T03:06:51.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.520+0000 7f08767fc700 1 --2- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f08600367e0 0x7f0860038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T03:06:51.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.921+0000 7f08767fc700 1 -- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f08600367e0 msgr2=0x7f0860038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3609017287 2024-03-19T03:06:51.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:45.921+0000 7f08767fc700 1 --2- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f08600367e0 0x7f0860038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T03:06:51.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:46.722+0000 7f08767fc700 1 -- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f08600367e0 msgr2=0x7f0860038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3609017287 2024-03-19T03:06:51.334 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:46.722+0000 7f08767fc700 1 --2- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f08600367e0 0x7f0860038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T03:06:51.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:48.323+0000 7f08767fc700 1 -- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f08600367e0 msgr2=0x7f0860038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3609017287 2024-03-19T03:06:51.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:48.323+0000 7f08767fc700 1 --2- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f08600367e0 0x7f0860038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-19T03:06:51.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:49.797+0000 7f085f7fe700 1 -- 172.21.15.59:0/2405486810 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mgrmap(e 6) v1 ==== 43135+0+0 (secure 0 0 0) 0x7f086c02e790 con 0x7f0878076c30 2024-03-19T03:06:51.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:49.797+0000 7f085f7fe700 1 -- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f08600367e0 msgr2=0x7f0860038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T03:06:51.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:49.797+0000 7f085f7fe700 1 --2- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f08600367e0 0x7f0860038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:51.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:50.797+0000 7f085f7fe700 1 -- 172.21.15.59:0/2405486810 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7f086c0243f0 con 0x7f0878076c30 2024-03-19T03:06:51.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:50.798+0000 7f085f7fe700 1 --2- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f0860039f50 0x7f086003c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:51.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:50.798+0000 7f085f7fe700 1 -- 172.21.15.59:0/2405486810 --> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f087804bed0 con 0x7f0860039f50 2024-03-19T03:06:51.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:50.798+0000 7f08767fc700 1 --2- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f0860039f50 0x7f086003c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:51.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:50.799+0000 7f08767fc700 1 --2- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f0860039f50 0x7f086003c330 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f08680039d0 tx=0x7f0868009350 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:51.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:50.799+0000 7f085f7fe700 1 -- 172.21.15.59:0/2405486810 <== mgr.14120 v2:172.21.15.59:6800/1573642037 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f087804bed0 con 0x7f0860039f50 2024-03-19T03:06:51.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:50.805+0000 7f087d26f700 1 -- 172.21.15.59:0/2405486810 --> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f0878106750 con 0x7f0860039f50 2024-03-19T03:06:51.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:50.805+0000 7f085f7fe700 1 -- 172.21.15.59:0/2405486810 <== mgr.14120 v2:172.21.15.59:6800/1573642037 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f0878106750 con 0x7f0860039f50 2024-03-19T03:06:51.335 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:50.806+0000 7f087d26f700 1 -- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f0860039f50 msgr2=0x7f086003c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:51.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:50.806+0000 7f087d26f700 1 --2- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f0860039f50 0x7f086003c330 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f08680039d0 tx=0x7f0868009350 comp rx=0 tx=0).stop 2024-03-19T03:06:51.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:50.806+0000 7f087d26f700 1 -- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0878076c30 msgr2=0x7f0878102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:51.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:50.806+0000 7f087d26f700 1 --2- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0878076c30 0x7f0878102c10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f086c003050 tx=0x7f086c009250 comp rx=0 tx=0).stop 2024-03-19T03:06:51.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:50.806+0000 7f087d26f700 1 -- 172.21.15.59:0/2405486810 shutdown_connections 2024-03-19T03:06:51.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:50.806+0000 7f087d26f700 1 --2- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f0860039f50 0x7f086003c330 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:51.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:50.806+0000 7f087d26f700 1 --2- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0878076c30 0x7f0878102c10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:51.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:50.806+0000 7f087d26f700 1 --2- 172.21.15.59:0/2405486810 >> [v2:172.21.15.59:6800/3609017287,v1:172.21.15.59:6801/3609017287] conn(0x7f08600367e0 0x7f0860038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:51.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:50.806+0000 7f087d26f700 1 -- 172.21.15.59:0/2405486810 >> 172.21.15.59:0/2405486810 conn(0x7f08780fcd50 msgr2=0x7f08780fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:51.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:50.806+0000 7f087d26f700 1 -- 172.21.15.59:0/2405486810 shutdown_connections 2024-03-19T03:06:51.336 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:50.806+0000 7f087d26f700 1 -- 172.21.15.59:0/2405486810 wait complete. 2024-03-19T03:06:51.336 INFO:teuthology.orchestra.run.smithi059.stdout:mgr epoch 5 is available 2024-03-19T03:06:51.336 INFO:teuthology.orchestra.run.smithi059.stdout:Setting orchestrator backend to cephadm... 2024-03-19T03:06:51.983 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:51 smithi059 ceph-mon[29133]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-19T03:06:51.983 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:51 smithi059 ceph-mon[29133]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-19T03:06:52.545 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.745+0000 7f529cc5b700 1 Processor -- start 2024-03-19T03:06:52.545 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.745+0000 7f529cc5b700 1 -- start start 2024-03-19T03:06:52.545 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.745+0000 7f529cc5b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5298102740 0x7f5298102b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:52.545 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.745+0000 7f529cc5b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5298103090 con 0x7f5298102740 2024-03-19T03:06:52.545 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.745+0000 7f529659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5298102740 0x7f5298102b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:52.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.745+0000 7f529659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5298102740 0x7f5298102b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48860/0 (socket says 172.21.15.59:48860) 2024-03-19T03:06:52.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.745+0000 7f529659c700 1 -- 172.21.15.59:0/2605832340 learned_addr learned my addr 172.21.15.59:0/2605832340 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:52.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.745+0000 7f529659c700 1 -- 172.21.15.59:0/2605832340 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52981031d0 con 0x7f5298102740 2024-03-19T03:06:52.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.746+0000 7f529659c700 1 --2- 172.21.15.59:0/2605832340 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5298102740 0x7f5298102b50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f52880098e0 tx=0x7f5288009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c252d3398e0ad2f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:52.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.746+0000 7f529559a700 1 -- 172.21.15.59:0/2605832340 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f528800b860 con 0x7f5298102740 2024-03-19T03:06:52.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.746+0000 7f529559a700 1 -- 172.21.15.59:0/2605832340 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f528800be50 con 0x7f5298102740 2024-03-19T03:06:52.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.746+0000 7f529559a700 1 -- 172.21.15.59:0/2605832340 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5288010380 con 0x7f5298102740 2024-03-19T03:06:52.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.746+0000 7f529cc5b700 1 -- 172.21.15.59:0/2605832340 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5298102740 msgr2=0x7f5298102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:52.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.746+0000 7f529cc5b700 1 --2- 172.21.15.59:0/2605832340 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5298102740 0x7f5298102b50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f52880098e0 tx=0x7f5288009250 comp rx=0 tx=0).stop 2024-03-19T03:06:52.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.747+0000 7f529cc5b700 1 -- 172.21.15.59:0/2605832340 shutdown_connections 2024-03-19T03:06:52.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.747+0000 7f529cc5b700 1 --2- 172.21.15.59:0/2605832340 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5298102740 0x7f5298102b50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:52.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.747+0000 7f529cc5b700 1 -- 172.21.15.59:0/2605832340 >> 172.21.15.59:0/2605832340 conn(0x7f52980fdea0 msgr2=0x7f52981002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:52.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.747+0000 7f529cc5b700 1 -- 172.21.15.59:0/2605832340 shutdown_connections 2024-03-19T03:06:52.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.747+0000 7f529cc5b700 1 -- 172.21.15.59:0/2605832340 wait complete. 2024-03-19T03:06:52.547 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.747+0000 7f529cc5b700 1 Processor -- start 2024-03-19T03:06:52.547 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.747+0000 7f529cc5b700 1 -- start start 2024-03-19T03:06:52.547 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.747+0000 7f529cc5b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5298102740 0x7f52981115b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:52.547 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.747+0000 7f529cc5b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5298111af0 con 0x7f5298102740 2024-03-19T03:06:52.547 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.747+0000 7f529659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5298102740 0x7f52981115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:52.547 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.747+0000 7f529659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5298102740 0x7f52981115b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48872/0 (socket says 172.21.15.59:48872) 2024-03-19T03:06:52.547 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.747+0000 7f529659c700 1 -- 172.21.15.59:0/1160239958 learned_addr learned my addr 172.21.15.59:0/1160239958 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:52.547 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.748+0000 7f529659c700 1 -- 172.21.15.59:0/1160239958 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52880094f0 con 0x7f5298102740 2024-03-19T03:06:52.547 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.748+0000 7f529659c700 1 --2- 172.21.15.59:0/1160239958 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5298102740 0x7f52981115b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f528800ba90 tx=0x7f5288009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:52.547 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.748+0000 7f5286ffd700 1 -- 172.21.15.59:0/1160239958 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52880107c0 con 0x7f5298102740 2024-03-19T03:06:52.547 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.748+0000 7f5286ffd700 1 -- 172.21.15.59:0/1160239958 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5288010db0 con 0x7f5298102740 2024-03-19T03:06:52.547 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.748+0000 7f529cc5b700 1 -- 172.21.15.59:0/1160239958 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5298111cf0 con 0x7f5298102740 2024-03-19T03:06:52.547 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.748+0000 7f5286ffd700 1 -- 172.21.15.59:0/1160239958 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f528801add0 con 0x7f5298102740 2024-03-19T03:06:52.547 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.748+0000 7f529cc5b700 1 -- 172.21.15.59:0/1160239958 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5298112860 con 0x7f5298102740 2024-03-19T03:06:52.548 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.749+0000 7f5286ffd700 1 -- 172.21.15.59:0/1160239958 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7f528802e440 con 0x7f5298102740 2024-03-19T03:06:52.548 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.749+0000 7f5286ffd700 1 --2- 172.21.15.59:0/1160239958 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f5280036660 0x7f5280038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:52.548 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.749+0000 7f5286ffd700 1 -- 172.21.15.59:0/1160239958 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f528804a5c0 con 0x7f5298102740 2024-03-19T03:06:52.548 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.749+0000 7f5295d9b700 1 --2- 172.21.15.59:0/1160239958 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f5280036660 0x7f5280038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:52.548 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.749+0000 7f529cc5b700 1 -- 172.21.15.59:0/1160239958 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f529805f5f0 con 0x7f5298102740 2024-03-19T03:06:52.548 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.750+0000 7f5295d9b700 1 --2- 172.21.15.59:0/1160239958 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f5280036660 0x7f5280038b10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f528c009990 tx=0x7f528c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:52.548 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.754+0000 7f5286ffd700 1 -- 172.21.15.59:0/1160239958 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f5288023b40 con 0x7f5298102740 2024-03-19T03:06:52.548 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.849+0000 7f5286ffd700 1 -- 172.21.15.59:0/1160239958 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f528802e790 con 0x7f5298102740 2024-03-19T03:06:52.548 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.960+0000 7f529cc5b700 1 -- 172.21.15.59:0/1160239958 --> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f52980ff120 con 0x7f5280036660 2024-03-19T03:06:52.548 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.969+0000 7f5286ffd700 1 -- 172.21.15.59:0/1160239958 <== mgr.14120 v2:172.21.15.59:6800/1573642037 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f52980ff120 con 0x7f5280036660 2024-03-19T03:06:52.548 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.971+0000 7f529cc5b700 1 -- 172.21.15.59:0/1160239958 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f5280036660 msgr2=0x7f5280038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:52.548 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.971+0000 7f529cc5b700 1 --2- 172.21.15.59:0/1160239958 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f5280036660 0x7f5280038b10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f528c009990 tx=0x7f528c006e40 comp rx=0 tx=0).stop 2024-03-19T03:06:52.548 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.971+0000 7f529cc5b700 1 -- 172.21.15.59:0/1160239958 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5298102740 msgr2=0x7f52981115b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:52.549 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.971+0000 7f529cc5b700 1 --2- 172.21.15.59:0/1160239958 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5298102740 0x7f52981115b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f528800ba90 tx=0x7f5288009250 comp rx=0 tx=0).stop 2024-03-19T03:06:52.549 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.971+0000 7f529cc5b700 1 -- 172.21.15.59:0/1160239958 shutdown_connections 2024-03-19T03:06:52.549 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.971+0000 7f529cc5b700 1 --2- 172.21.15.59:0/1160239958 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f5280036660 0x7f5280038b10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:52.549 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.971+0000 7f529cc5b700 1 --2- 172.21.15.59:0/1160239958 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5298102740 0x7f52981115b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:52.549 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.971+0000 7f529cc5b700 1 -- 172.21.15.59:0/1160239958 >> 172.21.15.59:0/1160239958 conn(0x7f52980fdea0 msgr2=0x7f52980fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:52.549 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.971+0000 7f529cc5b700 1 -- 172.21.15.59:0/1160239958 shutdown_connections 2024-03-19T03:06:52.549 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:51.971+0000 7f529cc5b700 1 -- 172.21.15.59:0/1160239958 wait complete. 2024-03-19T03:06:52.956 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:52 smithi059 ceph-mon[29133]: mgrmap e8: smithi059.azpecc(active, since 2s) 2024-03-19T03:06:52.956 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:52 smithi059 ceph-mon[29133]: [19/Mar/2024:03:06:51] ENGINE Bus STARTING 2024-03-19T03:06:52.956 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:52 smithi059 ceph-mon[29133]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:06:52.956 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:52 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' 2024-03-19T03:06:52.957 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:52 smithi059 ceph-mon[29133]: [19/Mar/2024:03:06:51] ENGINE Serving on https://172.21.15.59:7150 2024-03-19T03:06:52.957 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:52 smithi059 ceph-mon[29133]: [19/Mar/2024:03:06:51] ENGINE Bus STARTED 2024-03-19T03:06:52.957 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:52 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' 2024-03-19T03:06:52.957 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:52 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:06:53.742 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-19T03:06:53.743 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.983+0000 7f5216d4a700 1 Processor -- start 2024-03-19T03:06:53.743 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.983+0000 7f5216d4a700 1 -- start start 2024-03-19T03:06:53.743 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.983+0000 7f5216d4a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5210102730 0x7f5210102b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:53.743 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.983+0000 7f5216d4a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5210103080 con 0x7f5210102730 2024-03-19T03:06:53.743 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.983+0000 7f5214ae6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5210102730 0x7f5210102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:53.743 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.983+0000 7f5214ae6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5210102730 0x7f5210102b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48888/0 (socket says 172.21.15.59:48888) 2024-03-19T03:06:53.743 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.983+0000 7f5214ae6700 1 -- 172.21.15.59:0/416189016 learned_addr learned my addr 172.21.15.59:0/416189016 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:53.743 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.983+0000 7f5214ae6700 1 -- 172.21.15.59:0/416189016 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52101031c0 con 0x7f5210102730 2024-03-19T03:06:53.743 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.983+0000 7f5214ae6700 1 --2- 172.21.15.59:0/416189016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5210102730 0x7f5210102b40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f52040098e0 tx=0x7f5204009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f75802a98b1042 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:53.743 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.984+0000 7f520f7fe700 1 -- 172.21.15.59:0/416189016 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f520400b860 con 0x7f5210102730 2024-03-19T03:06:53.743 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.984+0000 7f520f7fe700 1 -- 172.21.15.59:0/416189016 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f520400be50 con 0x7f5210102730 2024-03-19T03:06:53.744 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.984+0000 7f520f7fe700 1 -- 172.21.15.59:0/416189016 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5204010380 con 0x7f5210102730 2024-03-19T03:06:53.744 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.984+0000 7f5216d4a700 1 -- 172.21.15.59:0/416189016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5210102730 msgr2=0x7f5210102b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:53.744 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.984+0000 7f5216d4a700 1 --2- 172.21.15.59:0/416189016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5210102730 0x7f5210102b40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f52040098e0 tx=0x7f5204009250 comp rx=0 tx=0).stop 2024-03-19T03:06:53.744 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.984+0000 7f5216d4a700 1 -- 172.21.15.59:0/416189016 shutdown_connections 2024-03-19T03:06:53.744 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.984+0000 7f5216d4a700 1 --2- 172.21.15.59:0/416189016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5210102730 0x7f5210102b40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:53.744 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.984+0000 7f5216d4a700 1 -- 172.21.15.59:0/416189016 >> 172.21.15.59:0/416189016 conn(0x7f52100fdf10 msgr2=0x7f5210100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:53.744 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.984+0000 7f5216d4a700 1 -- 172.21.15.59:0/416189016 shutdown_connections 2024-03-19T03:06:53.744 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.984+0000 7f5216d4a700 1 -- 172.21.15.59:0/416189016 wait complete. 2024-03-19T03:06:53.744 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.985+0000 7f5216d4a700 1 Processor -- start 2024-03-19T03:06:53.744 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.985+0000 7f5216d4a700 1 -- start start 2024-03-19T03:06:53.744 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.985+0000 7f5216d4a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5210102730 0x7f5210078bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:53.744 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.985+0000 7f5216d4a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5210079110 con 0x7f5210102730 2024-03-19T03:06:53.744 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.985+0000 7f5214ae6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5210102730 0x7f5210078bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:53.744 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.985+0000 7f5214ae6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5210102730 0x7f5210078bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48896/0 (socket says 172.21.15.59:48896) 2024-03-19T03:06:53.744 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.985+0000 7f5214ae6700 1 -- 172.21.15.59:0/1797159857 learned_addr learned my addr 172.21.15.59:0/1797159857 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:53.745 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.986+0000 7f5214ae6700 1 -- 172.21.15.59:0/1797159857 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52040094f0 con 0x7f5210102730 2024-03-19T03:06:53.745 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.986+0000 7f5214ae6700 1 --2- 172.21.15.59:0/1797159857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5210102730 0x7f5210078bd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f520400b440 tx=0x7f5204009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:53.745 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.986+0000 7f520d7fa700 1 -- 172.21.15.59:0/1797159857 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5204010760 con 0x7f5210102730 2024-03-19T03:06:53.745 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.986+0000 7f520d7fa700 1 -- 172.21.15.59:0/1797159857 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5204010d50 con 0x7f5210102730 2024-03-19T03:06:53.745 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.986+0000 7f5216d4a700 1 -- 172.21.15.59:0/1797159857 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5210077180 con 0x7f5210102730 2024-03-19T03:06:53.745 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.986+0000 7f520d7fa700 1 -- 172.21.15.59:0/1797159857 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f520401adc0 con 0x7f5210102730 2024-03-19T03:06:53.745 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.986+0000 7f5216d4a700 1 -- 172.21.15.59:0/1797159857 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5210077670 con 0x7f5210102730 2024-03-19T03:06:53.745 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.987+0000 7f520d7fa700 1 -- 172.21.15.59:0/1797159857 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f52040193c0 con 0x7f5210102730 2024-03-19T03:06:53.745 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.987+0000 7f520d7fa700 1 --2- 172.21.15.59:0/1797159857 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f52000367e0 0x7f5200038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:53.745 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.987+0000 7f5216d4a700 1 -- 172.21.15.59:0/1797159857 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f521004bed0 con 0x7f5210102730 2024-03-19T03:06:53.745 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.987+0000 7f520d7fa700 1 -- 172.21.15.59:0/1797159857 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5204049f40 con 0x7f5210102730 2024-03-19T03:06:53.745 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.987+0000 7f520ffff700 1 --2- 172.21.15.59:0/1797159857 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f52000367e0 0x7f5200038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:53.745 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.987+0000 7f520ffff700 1 --2- 172.21.15.59:0/1797159857 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f52000367e0 0x7f5200038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f51fc009990 tx=0x7f51fc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:53.745 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:52.992+0000 7f520d7fa700 1 -- 172.21.15.59:0/1797159857 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f5204023940 con 0x7f5210102730 2024-03-19T03:06:53.746 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:53.186+0000 7f5216d4a700 1 -- 172.21.15.59:0/1797159857 --> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f5210077d70 con 0x7f52000367e0 2024-03-19T03:06:53.746 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:53.187+0000 7f520d7fa700 1 -- 172.21.15.59:0/1797159857 <== mgr.14120 v2:172.21.15.59:6800/1573642037 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f5210077d70 con 0x7f52000367e0 2024-03-19T03:06:53.746 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:53.189+0000 7f5216d4a700 1 -- 172.21.15.59:0/1797159857 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f52000367e0 msgr2=0x7f5200038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:53.746 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:53.189+0000 7f5216d4a700 1 --2- 172.21.15.59:0/1797159857 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f52000367e0 0x7f5200038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f51fc009990 tx=0x7f51fc006e40 comp rx=0 tx=0).stop 2024-03-19T03:06:53.746 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:53.189+0000 7f5216d4a700 1 -- 172.21.15.59:0/1797159857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5210102730 msgr2=0x7f5210078bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:53.746 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:53.189+0000 7f5216d4a700 1 --2- 172.21.15.59:0/1797159857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5210102730 0x7f5210078bd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f520400b440 tx=0x7f5204009250 comp rx=0 tx=0).stop 2024-03-19T03:06:53.746 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:53.189+0000 7f5216d4a700 1 -- 172.21.15.59:0/1797159857 shutdown_connections 2024-03-19T03:06:53.746 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:53.189+0000 7f5216d4a700 1 --2- 172.21.15.59:0/1797159857 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f52000367e0 0x7f5200038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:53.746 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:53.189+0000 7f5216d4a700 1 --2- 172.21.15.59:0/1797159857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5210102730 0x7f5210078bd0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:53.746 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:53.189+0000 7f5216d4a700 1 -- 172.21.15.59:0/1797159857 >> 172.21.15.59:0/1797159857 conn(0x7f52100fdf10 msgr2=0x7f52100feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:53.746 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:53.189+0000 7f5216d4a700 1 -- 172.21.15.59:0/1797159857 shutdown_connections 2024-03-19T03:06:53.746 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:53.189+0000 7f5216d4a700 1 -- 172.21.15.59:0/1797159857 wait complete. 2024-03-19T03:06:53.746 INFO:teuthology.orchestra.run.smithi059.stdout:Generating ssh key... 2024-03-19T03:06:54.038 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:53 smithi059 ceph-mon[29133]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:06:55.549 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.138+0000 7ff43eff8700 1 Processor -- start 2024-03-19T03:06:55.549 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.138+0000 7ff43eff8700 1 -- start start 2024-03-19T03:06:55.549 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.138+0000 7ff43eff8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff438102c40 0x7ff438103050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:55.550 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.138+0000 7ff43eff8700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff438103590 con 0x7ff438102c40 2024-03-19T03:06:55.550 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.138+0000 7ff43cd94700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff438102c40 0x7ff438103050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:55.550 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.138+0000 7ff43cd94700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff438102c40 0x7ff438103050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51232/0 (socket says 172.21.15.59:51232) 2024-03-19T03:06:55.550 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.138+0000 7ff43cd94700 1 -- 172.21.15.59:0/1782504339 learned_addr learned my addr 172.21.15.59:0/1782504339 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:55.550 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.138+0000 7ff43cd94700 1 -- 172.21.15.59:0/1782504339 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4381036d0 con 0x7ff438102c40 2024-03-19T03:06:55.550 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.138+0000 7ff43cd94700 1 --2- 172.21.15.59:0/1782504339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff438102c40 0x7ff438103050 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff42c0098e0 tx=0x7ff42c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd86dc678bc861c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:55.550 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.139+0000 7ff4377fe700 1 -- 172.21.15.59:0/1782504339 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff42c00b860 con 0x7ff438102c40 2024-03-19T03:06:55.550 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.139+0000 7ff4377fe700 1 -- 172.21.15.59:0/1782504339 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff42c00be50 con 0x7ff438102c40 2024-03-19T03:06:55.550 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.139+0000 7ff4377fe700 1 -- 172.21.15.59:0/1782504339 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff42c010380 con 0x7ff438102c40 2024-03-19T03:06:55.550 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.139+0000 7ff43eff8700 1 -- 172.21.15.59:0/1782504339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff438102c40 msgr2=0x7ff438103050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:55.550 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.139+0000 7ff43eff8700 1 --2- 172.21.15.59:0/1782504339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff438102c40 0x7ff438103050 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff42c0098e0 tx=0x7ff42c009250 comp rx=0 tx=0).stop 2024-03-19T03:06:55.550 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.139+0000 7ff43eff8700 1 -- 172.21.15.59:0/1782504339 shutdown_connections 2024-03-19T03:06:55.550 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.139+0000 7ff43eff8700 1 --2- 172.21.15.59:0/1782504339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff438102c40 0x7ff438103050 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:55.551 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.139+0000 7ff43eff8700 1 -- 172.21.15.59:0/1782504339 >> 172.21.15.59:0/1782504339 conn(0x7ff4380fc560 msgr2=0x7ff4380fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:55.551 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.139+0000 7ff43eff8700 1 -- 172.21.15.59:0/1782504339 shutdown_connections 2024-03-19T03:06:55.551 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.139+0000 7ff43eff8700 1 -- 172.21.15.59:0/1782504339 wait complete. 2024-03-19T03:06:55.551 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.140+0000 7ff43eff8700 1 Processor -- start 2024-03-19T03:06:55.551 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.140+0000 7ff43eff8700 1 -- start start 2024-03-19T03:06:55.551 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.140+0000 7ff43eff8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff438102c40 0x7ff4381013f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:55.551 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.140+0000 7ff43eff8700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4381142e0 con 0x7ff438102c40 2024-03-19T03:06:55.551 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.140+0000 7ff43cd94700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff438102c40 0x7ff4381013f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:55.551 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.140+0000 7ff43cd94700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff438102c40 0x7ff4381013f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51236/0 (socket says 172.21.15.59:51236) 2024-03-19T03:06:55.551 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.140+0000 7ff43cd94700 1 -- 172.21.15.59:0/203801517 learned_addr learned my addr 172.21.15.59:0/203801517 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:55.551 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.141+0000 7ff43cd94700 1 -- 172.21.15.59:0/203801517 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff42c0094f0 con 0x7ff438102c40 2024-03-19T03:06:55.551 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.141+0000 7ff43cd94700 1 --2- 172.21.15.59:0/203801517 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff438102c40 0x7ff4381013f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff42c00ba90 tx=0x7ff42c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:55.551 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.141+0000 7ff4357fa700 1 -- 172.21.15.59:0/203801517 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff42c0107c0 con 0x7ff438102c40 2024-03-19T03:06:55.551 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.141+0000 7ff4357fa700 1 -- 172.21.15.59:0/203801517 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff42c010db0 con 0x7ff438102c40 2024-03-19T03:06:55.552 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.141+0000 7ff43eff8700 1 -- 172.21.15.59:0/203801517 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff438114420 con 0x7ff438102c40 2024-03-19T03:06:55.552 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.141+0000 7ff4357fa700 1 -- 172.21.15.59:0/203801517 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff42c01add0 con 0x7ff438102c40 2024-03-19T03:06:55.552 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.141+0000 7ff43eff8700 1 -- 172.21.15.59:0/203801517 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4381148b0 con 0x7ff438102c40 2024-03-19T03:06:55.552 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.142+0000 7ff4357fa700 1 -- 172.21.15.59:0/203801517 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7ff42c02e440 con 0x7ff438102c40 2024-03-19T03:06:55.552 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.142+0000 7ff4357fa700 1 --2- 172.21.15.59:0/203801517 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7ff4280367e0 0x7ff428038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:55.552 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.142+0000 7ff43eff8700 1 -- 172.21.15.59:0/203801517 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff43804bed0 con 0x7ff438102c40 2024-03-19T03:06:55.552 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.142+0000 7ff4357fa700 1 -- 172.21.15.59:0/203801517 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff42c04a5a0 con 0x7ff438102c40 2024-03-19T03:06:55.552 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.142+0000 7ff437fff700 1 --2- 172.21.15.59:0/203801517 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7ff4280367e0 0x7ff428038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:55.552 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.142+0000 7ff437fff700 1 --2- 172.21.15.59:0/203801517 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7ff4280367e0 0x7ff428038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff424009990 tx=0x7ff424006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:55.552 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.147+0000 7ff4357fa700 1 -- 172.21.15.59:0/203801517 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7ff42c018940 con 0x7ff438102c40 2024-03-19T03:06:55.552 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.339+0000 7ff43eff8700 1 -- 172.21.15.59:0/203801517 --> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4380fd7f0 con 0x7ff4280367e0 2024-03-19T03:06:55.552 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.959+0000 7ff4357fa700 1 -- 172.21.15.59:0/203801517 <== mgr.14120 v2:172.21.15.59:6800/1573642037 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff4380fd7f0 con 0x7ff4280367e0 2024-03-19T03:06:55.552 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.961+0000 7ff43eff8700 1 -- 172.21.15.59:0/203801517 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7ff4280367e0 msgr2=0x7ff428038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:55.552 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.961+0000 7ff43eff8700 1 --2- 172.21.15.59:0/203801517 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7ff4280367e0 0x7ff428038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff424009990 tx=0x7ff424006e40 comp rx=0 tx=0).stop 2024-03-19T03:06:55.553 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.961+0000 7ff43eff8700 1 -- 172.21.15.59:0/203801517 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff438102c40 msgr2=0x7ff4381013f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:55.553 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.961+0000 7ff43eff8700 1 --2- 172.21.15.59:0/203801517 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff438102c40 0x7ff4381013f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff42c00ba90 tx=0x7ff42c009250 comp rx=0 tx=0).stop 2024-03-19T03:06:55.553 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.962+0000 7ff43eff8700 1 -- 172.21.15.59:0/203801517 shutdown_connections 2024-03-19T03:06:55.553 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.962+0000 7ff43eff8700 1 --2- 172.21.15.59:0/203801517 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7ff4280367e0 0x7ff428038c90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:55.553 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.962+0000 7ff43eff8700 1 --2- 172.21.15.59:0/203801517 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff438102c40 0x7ff4381013f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:55.553 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.962+0000 7ff43eff8700 1 -- 172.21.15.59:0/203801517 >> 172.21.15.59:0/203801517 conn(0x7ff4380fc560 msgr2=0x7ff4380fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:55.553 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.962+0000 7ff43eff8700 1 -- 172.21.15.59:0/203801517 shutdown_connections 2024-03-19T03:06:55.553 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:54.962+0000 7ff43eff8700 1 -- 172.21.15.59:0/203801517 wait complete. 2024-03-19T03:06:56.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:55 smithi059 ceph-mon[29133]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:06:56.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:55 smithi059 ceph-mon[29133]: Generating ssh key... 2024-03-19T03:06:56.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:55 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' 2024-03-19T03:06:56.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:55 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' 2024-03-19T03:06:56.665 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOlnUuONg0bL3nPOacT9ojXYWnGrvA7ywhiXYrtKhCOhO4O/V1DrisuAIKAWo4HBCZAwlm2i+RzoqZiuxgXWRVwOFfV5s4vTEtCRr3b0gx61HfJHz5tatZwKSRbSxNYA63NEdDfRczfboTZFXFOvYoX4680X15HU3I1oYgU84yJ6raLH0AzELJiMrmKfWEe4dszmwgRtzW/qNXETin8TTKlbyj8s2QEwwyJw4YRRpXQfxyit1NpdkOvSybiXhbHGSkX9CK2DSdZi4uz9XrQ9aHsaUrUgRXoZ6MMv6T8HD0tGEDs2GMAfhwCMFEPn3ydTEANZ1dkp9SwhpzqM6widkBMSXinOYICxyJcn+BLYex3+lMmfqN90+4EzoFab7j3aJpptof5uNTbR1XnTx3GVw3wdvFaFLSNuLlviciPpqzjHVbAOCJ3EU2ilGaFHBMKLFaaGcc20iAtKx87Ot0DP1XjEmB6j1brlVI/k1zerfVyMUf+oVutj65x+TzzkJbGnE= ceph-8c8ae530-e59d-11ee-95c9-87774f69a715 2024-03-19T03:06:56.665 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.917+0000 7fc0aaac4700 1 Processor -- start 2024-03-19T03:06:56.665 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.917+0000 7fc0aaac4700 1 -- start start 2024-03-19T03:06:56.665 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.917+0000 7fc0aaac4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0a4078ca0 0x7fc0a4077200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:56.665 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.917+0000 7fc0aaac4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a40790b0 con 0x7fc0a4078ca0 2024-03-19T03:06:56.665 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.917+0000 7fc0a8860700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0a4078ca0 0x7fc0a4077200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:56.665 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.917+0000 7fc0a8860700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0a4078ca0 0x7fc0a4077200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51240/0 (socket says 172.21.15.59:51240) 2024-03-19T03:06:56.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.917+0000 7fc0a8860700 1 -- 172.21.15.59:0/2858337043 learned_addr learned my addr 172.21.15.59:0/2858337043 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:56.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.917+0000 7fc0a8860700 1 -- 172.21.15.59:0/2858337043 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a4077740 con 0x7fc0a4078ca0 2024-03-19T03:06:56.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.918+0000 7fc0a8860700 1 --2- 172.21.15.59:0/2858337043 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0a4078ca0 0x7fc0a4077200 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc0980098e0 tx=0x7fc098009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83247c9a94516639 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:56.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.918+0000 7fc0a37fe700 1 -- 172.21.15.59:0/2858337043 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc09800b860 con 0x7fc0a4078ca0 2024-03-19T03:06:56.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.918+0000 7fc0a37fe700 1 -- 172.21.15.59:0/2858337043 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fc09800be50 con 0x7fc0a4078ca0 2024-03-19T03:06:56.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.918+0000 7fc0a37fe700 1 -- 172.21.15.59:0/2858337043 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc098010380 con 0x7fc0a4078ca0 2024-03-19T03:06:56.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.918+0000 7fc0aaac4700 1 -- 172.21.15.59:0/2858337043 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0a4078ca0 msgr2=0x7fc0a4077200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:56.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.918+0000 7fc0aaac4700 1 --2- 172.21.15.59:0/2858337043 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0a4078ca0 0x7fc0a4077200 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc0980098e0 tx=0x7fc098009250 comp rx=0 tx=0).stop 2024-03-19T03:06:56.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.918+0000 7fc0aaac4700 1 -- 172.21.15.59:0/2858337043 shutdown_connections 2024-03-19T03:06:56.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.918+0000 7fc0aaac4700 1 --2- 172.21.15.59:0/2858337043 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0a4078ca0 0x7fc0a4077200 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:56.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.918+0000 7fc0aaac4700 1 -- 172.21.15.59:0/2858337043 >> 172.21.15.59:0/2858337043 conn(0x7fc0a40fdee0 msgr2=0x7fc0a41002f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:56.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.918+0000 7fc0aaac4700 1 -- 172.21.15.59:0/2858337043 shutdown_connections 2024-03-19T03:06:56.666 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.918+0000 7fc0aaac4700 1 -- 172.21.15.59:0/2858337043 wait complete. 2024-03-19T03:06:56.667 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.919+0000 7fc0aaac4700 1 Processor -- start 2024-03-19T03:06:56.667 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.919+0000 7fc0aaac4700 1 -- start start 2024-03-19T03:06:56.667 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.919+0000 7fc0aaac4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0a4078ca0 0x7fc0a41137d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:56.667 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.919+0000 7fc0aaac4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a4113d10 con 0x7fc0a4078ca0 2024-03-19T03:06:56.669 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.919+0000 7fc0a8860700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0a4078ca0 0x7fc0a41137d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:56.669 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.919+0000 7fc0a8860700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0a4078ca0 0x7fc0a41137d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51248/0 (socket says 172.21.15.59:51248) 2024-03-19T03:06:56.669 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.919+0000 7fc0a8860700 1 -- 172.21.15.59:0/1969922038 learned_addr learned my addr 172.21.15.59:0/1969922038 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:06:56.669 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.920+0000 7fc0a8860700 1 -- 172.21.15.59:0/1969922038 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0980094f0 con 0x7fc0a4078ca0 2024-03-19T03:06:56.669 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.920+0000 7fc0a8860700 1 --2- 172.21.15.59:0/1969922038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0a4078ca0 0x7fc0a41137d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc098006fd0 tx=0x7fc098009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:56.669 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.920+0000 7fc0a17fa700 1 -- 172.21.15.59:0/1969922038 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc098010760 con 0x7fc0a4078ca0 2024-03-19T03:06:56.669 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.920+0000 7fc0a17fa700 1 -- 172.21.15.59:0/1969922038 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fc098010d50 con 0x7fc0a4078ca0 2024-03-19T03:06:56.669 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.920+0000 7fc0aaac4700 1 -- 172.21.15.59:0/1969922038 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a4113f10 con 0x7fc0a4078ca0 2024-03-19T03:06:56.669 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.920+0000 7fc0a17fa700 1 -- 172.21.15.59:0/1969922038 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc09801ae10 con 0x7fc0a4078ca0 2024-03-19T03:06:56.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.920+0000 7fc0aaac4700 1 -- 172.21.15.59:0/1969922038 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a4114a80 con 0x7fc0a4078ca0 2024-03-19T03:06:56.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.921+0000 7fc0a17fa700 1 -- 172.21.15.59:0/1969922038 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fc098019410 con 0x7fc0a4078ca0 2024-03-19T03:06:56.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.921+0000 7fc0aaac4700 1 -- 172.21.15.59:0/1969922038 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a405f5f0 con 0x7fc0a4078ca0 2024-03-19T03:06:56.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.921+0000 7fc0a17fa700 1 --2- 172.21.15.59:0/1969922038 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fc094036720 0x7fc094038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:06:56.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.921+0000 7fc0a17fa700 1 -- 172.21.15.59:0/1969922038 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc098049fc0 con 0x7fc0a4078ca0 2024-03-19T03:06:56.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.921+0000 7fc0a3fff700 1 --2- 172.21.15.59:0/1969922038 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fc094036720 0x7fc094038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:06:56.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.921+0000 7fc0a3fff700 1 --2- 172.21.15.59:0/1969922038 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fc094036720 0x7fc094038bd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc090009990 tx=0x7fc090006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:06:56.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:55.926+0000 7fc0a17fa700 1 -- 172.21.15.59:0/1969922038 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fc098023710 con 0x7fc0a4078ca0 2024-03-19T03:06:56.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:56.117+0000 7fc0aaac4700 1 -- 172.21.15.59:0/1969922038 --> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0a40ff260 con 0x7fc094036720 2024-03-19T03:06:56.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:56.118+0000 7fc0a17fa700 1 -- 172.21.15.59:0/1969922038 <== mgr.14120 v2:172.21.15.59:6800/1573642037 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fc0a40ff260 con 0x7fc094036720 2024-03-19T03:06:56.670 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:56.120+0000 7fc0aaac4700 1 -- 172.21.15.59:0/1969922038 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fc094036720 msgr2=0x7fc094038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:56.671 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:56.120+0000 7fc0aaac4700 1 --2- 172.21.15.59:0/1969922038 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fc094036720 0x7fc094038bd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc090009990 tx=0x7fc090006e40 comp rx=0 tx=0).stop 2024-03-19T03:06:56.671 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:56.120+0000 7fc0aaac4700 1 -- 172.21.15.59:0/1969922038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0a4078ca0 msgr2=0x7fc0a41137d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:06:56.671 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:56.120+0000 7fc0aaac4700 1 --2- 172.21.15.59:0/1969922038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0a4078ca0 0x7fc0a41137d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc098006fd0 tx=0x7fc098009250 comp rx=0 tx=0).stop 2024-03-19T03:06:56.671 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:56.120+0000 7fc0aaac4700 1 -- 172.21.15.59:0/1969922038 shutdown_connections 2024-03-19T03:06:56.671 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:56.120+0000 7fc0aaac4700 1 --2- 172.21.15.59:0/1969922038 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fc094036720 0x7fc094038bd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:56.671 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:56.120+0000 7fc0aaac4700 1 --2- 172.21.15.59:0/1969922038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0a4078ca0 0x7fc0a41137d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:06:56.671 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:56.120+0000 7fc0aaac4700 1 -- 172.21.15.59:0/1969922038 >> 172.21.15.59:0/1969922038 conn(0x7fc0a40fdee0 msgr2=0x7fc0a40fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:06:56.671 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:56.120+0000 7fc0aaac4700 1 -- 172.21.15.59:0/1969922038 shutdown_connections 2024-03-19T03:06:56.671 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:56.120+0000 7fc0aaac4700 1 -- 172.21.15.59:0/1969922038 wait complete. 2024-03-19T03:06:56.671 INFO:teuthology.orchestra.run.smithi059.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-19T03:06:56.671 INFO:teuthology.orchestra.run.smithi059.stdout:Adding key to root@localhost authorized_keys... 2024-03-19T03:06:56.671 INFO:teuthology.orchestra.run.smithi059.stdout:Adding host smithi059... 2024-03-19T03:06:57.051 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:56 smithi059 ceph-mon[29133]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:06:58.504 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:58 smithi059 ceph-mon[29133]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi059", "addr": "172.21.15.59", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:06:58.504 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:06:58 smithi059 ceph-mon[29133]: Deploying cephadm binary to smithi059 2024-03-19T03:07:00.006 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Added host 'smithi059' with addr '172.21.15.59' 2024-03-19T03:07:00.006 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.076+0000 7fa21cbc8700 1 Processor -- start 2024-03-19T03:07:00.006 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.076+0000 7fa21cbc8700 1 -- start start 2024-03-19T03:07:00.006 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.076+0000 7fa21cbc8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa218102cc0 0x7fa2181030d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:00.006 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.076+0000 7fa21cbc8700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa218103610 con 0x7fa218102cc0 2024-03-19T03:07:00.006 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.077+0000 7fa21659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa218102cc0 0x7fa2181030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:00.006 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.077+0000 7fa21659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa218102cc0 0x7fa2181030d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51260/0 (socket says 172.21.15.59:51260) 2024-03-19T03:07:00.007 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.077+0000 7fa21659c700 1 -- 172.21.15.59:0/3684717679 learned_addr learned my addr 172.21.15.59:0/3684717679 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:00.007 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.077+0000 7fa21659c700 1 -- 172.21.15.59:0/3684717679 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa218103750 con 0x7fa218102cc0 2024-03-19T03:07:00.007 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.077+0000 7fa21659c700 1 --2- 172.21.15.59:0/3684717679 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa218102cc0 0x7fa2181030d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa20c0098e0 tx=0x7fa20c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96c3ebc8d64c694b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:00.007 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.077+0000 7fa21559a700 1 -- 172.21.15.59:0/3684717679 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa20c00b8f0 con 0x7fa218102cc0 2024-03-19T03:07:00.007 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.077+0000 7fa21559a700 1 -- 172.21.15.59:0/3684717679 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa20c011420 con 0x7fa218102cc0 2024-03-19T03:07:00.007 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.077+0000 7fa21559a700 1 -- 172.21.15.59:0/3684717679 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa20c010540 con 0x7fa218102cc0 2024-03-19T03:07:00.007 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.078+0000 7fa21cbc8700 1 -- 172.21.15.59:0/3684717679 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa218102cc0 msgr2=0x7fa2181030d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:00.007 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.078+0000 7fa21cbc8700 1 --2- 172.21.15.59:0/3684717679 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa218102cc0 0x7fa2181030d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa20c0098e0 tx=0x7fa20c009250 comp rx=0 tx=0).stop 2024-03-19T03:07:00.007 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.078+0000 7fa21cbc8700 1 -- 172.21.15.59:0/3684717679 shutdown_connections 2024-03-19T03:07:00.007 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.078+0000 7fa21cbc8700 1 --2- 172.21.15.59:0/3684717679 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa218102cc0 0x7fa2181030d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:00.007 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.078+0000 7fa21cbc8700 1 -- 172.21.15.59:0/3684717679 >> 172.21.15.59:0/3684717679 conn(0x7fa2180fc4d0 msgr2=0x7fa2180fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:00.007 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.078+0000 7fa21cbc8700 1 -- 172.21.15.59:0/3684717679 shutdown_connections 2024-03-19T03:07:00.007 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.078+0000 7fa21cbc8700 1 -- 172.21.15.59:0/3684717679 wait complete. 2024-03-19T03:07:00.008 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.078+0000 7fa21cbc8700 1 Processor -- start 2024-03-19T03:07:00.008 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.079+0000 7fa21cbc8700 1 -- start start 2024-03-19T03:07:00.008 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.079+0000 7fa21cbc8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa218102cc0 0x7fa2181137e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:00.008 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.079+0000 7fa21cbc8700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa218113d20 con 0x7fa218102cc0 2024-03-19T03:07:00.008 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.079+0000 7fa21659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa218102cc0 0x7fa2181137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:00.008 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.079+0000 7fa21659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa218102cc0 0x7fa2181137e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51274/0 (socket says 172.21.15.59:51274) 2024-03-19T03:07:00.008 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.079+0000 7fa21659c700 1 -- 172.21.15.59:0/4185112482 learned_addr learned my addr 172.21.15.59:0/4185112482 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:00.008 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.079+0000 7fa21659c700 1 -- 172.21.15.59:0/4185112482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa20c0094f0 con 0x7fa218102cc0 2024-03-19T03:07:00.008 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.079+0000 7fa21659c700 1 --2- 172.21.15.59:0/4185112482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa218102cc0 0x7fa2181137e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa20c010760 tx=0x7fa20c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:00.008 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.079+0000 7fa206ffd700 1 -- 172.21.15.59:0/4185112482 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa20c010800 con 0x7fa218102cc0 2024-03-19T03:07:00.008 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.079+0000 7fa206ffd700 1 -- 172.21.15.59:0/4185112482 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa20c010df0 con 0x7fa218102cc0 2024-03-19T03:07:00.008 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.079+0000 7fa21cbc8700 1 -- 172.21.15.59:0/4185112482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa218113f20 con 0x7fa218102cc0 2024-03-19T03:07:00.008 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.079+0000 7fa206ffd700 1 -- 172.21.15.59:0/4185112482 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa20c0193c0 con 0x7fa218102cc0 2024-03-19T03:07:00.008 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.079+0000 7fa21cbc8700 1 -- 172.21.15.59:0/4185112482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa218114a90 con 0x7fa218102cc0 2024-03-19T03:07:00.009 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.080+0000 7fa21cbc8700 1 -- 172.21.15.59:0/4185112482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa21805f5f0 con 0x7fa218102cc0 2024-03-19T03:07:00.009 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.081+0000 7fa206ffd700 1 -- 172.21.15.59:0/4185112482 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fa20c010960 con 0x7fa218102cc0 2024-03-19T03:07:00.009 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.081+0000 7fa206ffd700 1 --2- 172.21.15.59:0/4185112482 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fa200036790 0x7fa200038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:00.009 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.081+0000 7fa206ffd700 1 -- 172.21.15.59:0/4185112482 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa20c04a660 con 0x7fa218102cc0 2024-03-19T03:07:00.009 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.081+0000 7fa215d9b700 1 --2- 172.21.15.59:0/4185112482 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fa200036790 0x7fa200038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:00.009 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.081+0000 7fa215d9b700 1 --2- 172.21.15.59:0/4185112482 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fa200036790 0x7fa200038c40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa208009990 tx=0x7fa208006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:00.010 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.089+0000 7fa206ffd700 1 -- 172.21.15.59:0/4185112482 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fa20c022340 con 0x7fa218102cc0 2024-03-19T03:07:00.010 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:57.285+0000 7fa21cbc8700 1 -- 172.21.15.59:0/4185112482 --> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi059", "addr": "172.21.15.59", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2180fd8a0 con 0x7fa200036790 2024-03-19T03:07:00.010 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:59.368+0000 7fa206ffd700 1 -- 172.21.15.59:0/4185112482 <== mgr.14120 v2:172.21.15.59:6800/1573642037 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fa2180fd8a0 con 0x7fa200036790 2024-03-19T03:07:00.010 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:59.370+0000 7fa21cbc8700 1 -- 172.21.15.59:0/4185112482 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fa200036790 msgr2=0x7fa200038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:00.010 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:59.370+0000 7fa21cbc8700 1 --2- 172.21.15.59:0/4185112482 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fa200036790 0x7fa200038c40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa208009990 tx=0x7fa208006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:00.010 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:59.370+0000 7fa21cbc8700 1 -- 172.21.15.59:0/4185112482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa218102cc0 msgr2=0x7fa2181137e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:00.010 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:59.370+0000 7fa21cbc8700 1 --2- 172.21.15.59:0/4185112482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa218102cc0 0x7fa2181137e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa20c010760 tx=0x7fa20c009250 comp rx=0 tx=0).stop 2024-03-19T03:07:00.010 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:59.370+0000 7fa21cbc8700 1 -- 172.21.15.59:0/4185112482 shutdown_connections 2024-03-19T03:07:00.010 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:59.370+0000 7fa21cbc8700 1 --2- 172.21.15.59:0/4185112482 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fa200036790 0x7fa200038c40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:00.010 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:59.370+0000 7fa21cbc8700 1 --2- 172.21.15.59:0/4185112482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa218102cc0 0x7fa2181137e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:00.011 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:59.370+0000 7fa21cbc8700 1 -- 172.21.15.59:0/4185112482 >> 172.21.15.59:0/4185112482 conn(0x7fa2180fc4d0 msgr2=0x7fa2180fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:00.011 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:59.371+0000 7fa21cbc8700 1 -- 172.21.15.59:0/4185112482 shutdown_connections 2024-03-19T03:07:00.011 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:06:59.371+0000 7fa21cbc8700 1 -- 172.21.15.59:0/4185112482 wait complete. 2024-03-19T03:07:00.011 INFO:teuthology.orchestra.run.smithi059.stdout:Deploying mon service with default placement... 2024-03-19T03:07:00.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:00 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:00.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:00 smithi059 ceph-mon[29133]: Added host smithi059 2024-03-19T03:07:00.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:00 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:07:01.206 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-19T03:07:01.206 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.406+0000 7fe55e786700 1 Processor -- start 2024-03-19T03:07:01.206 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.406+0000 7fe55e786700 1 -- start start 2024-03-19T03:07:01.206 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.407+0000 7fe55e786700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe558102be0 0x7fe558102ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:01.206 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.407+0000 7fe55e786700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe558103530 con 0x7fe558102be0 2024-03-19T03:07:01.206 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.407+0000 7fe557fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe558102be0 0x7fe558102ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:01.206 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.407+0000 7fe557fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe558102be0 0x7fe558102ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51280/0 (socket says 172.21.15.59:51280) 2024-03-19T03:07:01.207 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.407+0000 7fe557fff700 1 -- 172.21.15.59:0/3859803530 learned_addr learned my addr 172.21.15.59:0/3859803530 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:01.207 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.407+0000 7fe557fff700 1 -- 172.21.15.59:0/3859803530 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe558103670 con 0x7fe558102be0 2024-03-19T03:07:01.207 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.407+0000 7fe557fff700 1 --2- 172.21.15.59:0/3859803530 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe558102be0 0x7fe558102ff0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe5480098e0 tx=0x7fe548009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f985fa07b05598 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:01.207 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.407+0000 7fe556ffd700 1 -- 172.21.15.59:0/3859803530 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe54800b860 con 0x7fe558102be0 2024-03-19T03:07:01.207 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.407+0000 7fe556ffd700 1 -- 172.21.15.59:0/3859803530 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fe54800be50 con 0x7fe558102be0 2024-03-19T03:07:01.207 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.407+0000 7fe556ffd700 1 -- 172.21.15.59:0/3859803530 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe548010380 con 0x7fe558102be0 2024-03-19T03:07:01.207 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.408+0000 7fe55e786700 1 -- 172.21.15.59:0/3859803530 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe558102be0 msgr2=0x7fe558102ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:01.207 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.408+0000 7fe55e786700 1 --2- 172.21.15.59:0/3859803530 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe558102be0 0x7fe558102ff0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe5480098e0 tx=0x7fe548009250 comp rx=0 tx=0).stop 2024-03-19T03:07:01.207 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.408+0000 7fe55e786700 1 -- 172.21.15.59:0/3859803530 shutdown_connections 2024-03-19T03:07:01.207 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.408+0000 7fe55e786700 1 --2- 172.21.15.59:0/3859803530 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe558102be0 0x7fe558102ff0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:01.207 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.408+0000 7fe55e786700 1 -- 172.21.15.59:0/3859803530 >> 172.21.15.59:0/3859803530 conn(0x7fe5580fc500 msgr2=0x7fe5580fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:01.207 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.408+0000 7fe55e786700 1 -- 172.21.15.59:0/3859803530 shutdown_connections 2024-03-19T03:07:01.207 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.408+0000 7fe55e786700 1 -- 172.21.15.59:0/3859803530 wait complete. 2024-03-19T03:07:01.207 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.409+0000 7fe55e786700 1 Processor -- start 2024-03-19T03:07:01.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.409+0000 7fe55e786700 1 -- start start 2024-03-19T03:07:01.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.409+0000 7fe55e786700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe558102be0 0x7fe558110440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:01.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.409+0000 7fe55e786700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe558110980 con 0x7fe558102be0 2024-03-19T03:07:01.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.409+0000 7fe557fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe558102be0 0x7fe558110440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:01.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.409+0000 7fe557fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe558102be0 0x7fe558110440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51284/0 (socket says 172.21.15.59:51284) 2024-03-19T03:07:01.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.409+0000 7fe557fff700 1 -- 172.21.15.59:0/1349394188 learned_addr learned my addr 172.21.15.59:0/1349394188 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:01.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.409+0000 7fe557fff700 1 -- 172.21.15.59:0/1349394188 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5480094f0 con 0x7fe558102be0 2024-03-19T03:07:01.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.410+0000 7fe557fff700 1 --2- 172.21.15.59:0/1349394188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe558102be0 0x7fe558110440 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe54800b440 tx=0x7fe548009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:01.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.410+0000 7fe554ff9700 1 -- 172.21.15.59:0/1349394188 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe548010760 con 0x7fe558102be0 2024-03-19T03:07:01.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.410+0000 7fe554ff9700 1 -- 172.21.15.59:0/1349394188 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fe548010d50 con 0x7fe558102be0 2024-03-19T03:07:01.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.410+0000 7fe55e786700 1 -- 172.21.15.59:0/1349394188 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe558110b80 con 0x7fe558102be0 2024-03-19T03:07:01.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.410+0000 7fe554ff9700 1 -- 172.21.15.59:0/1349394188 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe54801ae20 con 0x7fe558102be0 2024-03-19T03:07:01.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.410+0000 7fe55e786700 1 -- 172.21.15.59:0/1349394188 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5581116f0 con 0x7fe558102be0 2024-03-19T03:07:01.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.410+0000 7fe554ff9700 1 -- 172.21.15.59:0/1349394188 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fe54802e440 con 0x7fe558102be0 2024-03-19T03:07:01.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.411+0000 7fe554ff9700 1 --2- 172.21.15.59:0/1349394188 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fe5400367e0 0x7fe540038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:01.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.411+0000 7fe554ff9700 1 -- 172.21.15.59:0/1349394188 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe54804a560 con 0x7fe558102be0 2024-03-19T03:07:01.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.411+0000 7fe5577fe700 1 --2- 172.21.15.59:0/1349394188 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fe5400367e0 0x7fe540038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:01.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.411+0000 7fe55e786700 1 -- 172.21.15.59:0/1349394188 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe55805f5f0 con 0x7fe558102be0 2024-03-19T03:07:01.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.411+0000 7fe5577fe700 1 --2- 172.21.15.59:0/1349394188 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fe5400367e0 0x7fe540038c90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe54c009990 tx=0x7fe54c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:01.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.416+0000 7fe554ff9700 1 -- 172.21.15.59:0/1349394188 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fe548018530 con 0x7fe558102be0 2024-03-19T03:07:01.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.620+0000 7fe55e786700 1 -- 172.21.15.59:0/1349394188 --> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5580fd6f0 con 0x7fe5400367e0 2024-03-19T03:07:01.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.627+0000 7fe554ff9700 1 -- 172.21.15.59:0/1349394188 <== mgr.14120 v2:172.21.15.59:6800/1573642037 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fe5580fd6f0 con 0x7fe5400367e0 2024-03-19T03:07:01.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.628+0000 7fe55e786700 1 -- 172.21.15.59:0/1349394188 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fe5400367e0 msgr2=0x7fe540038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:01.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.628+0000 7fe55e786700 1 --2- 172.21.15.59:0/1349394188 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fe5400367e0 0x7fe540038c90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe54c009990 tx=0x7fe54c006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:01.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.629+0000 7fe55e786700 1 -- 172.21.15.59:0/1349394188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe558102be0 msgr2=0x7fe558110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:01.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.629+0000 7fe55e786700 1 --2- 172.21.15.59:0/1349394188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe558102be0 0x7fe558110440 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe54800b440 tx=0x7fe548009250 comp rx=0 tx=0).stop 2024-03-19T03:07:01.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.629+0000 7fe55e786700 1 -- 172.21.15.59:0/1349394188 shutdown_connections 2024-03-19T03:07:01.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.629+0000 7fe55e786700 1 --2- 172.21.15.59:0/1349394188 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fe5400367e0 0x7fe540038c90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:01.210 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.629+0000 7fe55e786700 1 --2- 172.21.15.59:0/1349394188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe558102be0 0x7fe558110440 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:01.210 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.629+0000 7fe55e786700 1 -- 172.21.15.59:0/1349394188 >> 172.21.15.59:0/1349394188 conn(0x7fe5580fc500 msgr2=0x7fe5580fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:01.210 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.629+0000 7fe55e786700 1 -- 172.21.15.59:0/1349394188 shutdown_connections 2024-03-19T03:07:01.210 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:00.629+0000 7fe55e786700 1 -- 172.21.15.59:0/1349394188 wait complete. 2024-03-19T03:07:01.210 INFO:teuthology.orchestra.run.smithi059.stdout:Deploying mgr service with default placement... 2024-03-19T03:07:01.767 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:01 smithi059 ceph-mon[29133]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:07:01.768 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:01 smithi059 ceph-mon[29133]: Saving service mon spec with placement count:5 2024-03-19T03:07:01.768 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:01 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:03.028 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:02 smithi059 ceph-mon[29133]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:07:03.028 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:02 smithi059 ceph-mon[29133]: Saving service mgr spec with placement count:2 2024-03-19T03:07:03.028 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:02 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:03.028 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:02 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:03.036 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-19T03:07:03.036 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.584+0000 7f2f9c247700 1 Processor -- start 2024-03-19T03:07:03.036 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.584+0000 7f2f9c247700 1 -- start start 2024-03-19T03:07:03.036 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.584+0000 7f2f9c247700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f94076b30 0x7f2f94107830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:03.036 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.584+0000 7f2f9c247700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f94103220 con 0x7f2f94076b30 2024-03-19T03:07:03.036 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.584+0000 7f2f99fe3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f94076b30 0x7f2f94107830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:03.036 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.584+0000 7f2f99fe3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f94076b30 0x7f2f94107830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51296/0 (socket says 172.21.15.59:51296) 2024-03-19T03:07:03.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.584+0000 7f2f99fe3700 1 -- 172.21.15.59:0/970869980 learned_addr learned my addr 172.21.15.59:0/970869980 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:03.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.585+0000 7f2f99fe3700 1 -- 172.21.15.59:0/970869980 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f94107d70 con 0x7f2f94076b30 2024-03-19T03:07:03.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.585+0000 7f2f99fe3700 1 --2- 172.21.15.59:0/970869980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f94076b30 0x7f2f94107830 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2f900098e0 tx=0x7f2f90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d74c1dea2f74275 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:03.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.585+0000 7f2f98fe1700 1 -- 172.21.15.59:0/970869980 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f9000b860 con 0x7f2f94076b30 2024-03-19T03:07:03.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.585+0000 7f2f98fe1700 1 -- 172.21.15.59:0/970869980 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2f9000be50 con 0x7f2f94076b30 2024-03-19T03:07:03.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.585+0000 7f2f98fe1700 1 -- 172.21.15.59:0/970869980 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f90010380 con 0x7f2f94076b30 2024-03-19T03:07:03.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.586+0000 7f2f9c247700 1 -- 172.21.15.59:0/970869980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f94076b30 msgr2=0x7f2f94107830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:03.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.586+0000 7f2f9c247700 1 --2- 172.21.15.59:0/970869980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f94076b30 0x7f2f94107830 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2f900098e0 tx=0x7f2f90009250 comp rx=0 tx=0).stop 2024-03-19T03:07:03.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.586+0000 7f2f9c247700 1 -- 172.21.15.59:0/970869980 shutdown_connections 2024-03-19T03:07:03.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.586+0000 7f2f9c247700 1 --2- 172.21.15.59:0/970869980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f94076b30 0x7f2f94107830 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:03.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.586+0000 7f2f9c247700 1 -- 172.21.15.59:0/970869980 >> 172.21.15.59:0/970869980 conn(0x7f2f940fce40 msgr2=0x7f2f940ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:03.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.586+0000 7f2f9c247700 1 -- 172.21.15.59:0/970869980 shutdown_connections 2024-03-19T03:07:03.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.586+0000 7f2f9c247700 1 -- 172.21.15.59:0/970869980 wait complete. 2024-03-19T03:07:03.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.586+0000 7f2f9c247700 1 Processor -- start 2024-03-19T03:07:03.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.586+0000 7f2f9c247700 1 -- start start 2024-03-19T03:07:03.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.586+0000 7f2f9c247700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f94076b30 0x7f2f941137d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:03.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.586+0000 7f2f9c247700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f94113d10 con 0x7f2f94076b30 2024-03-19T03:07:03.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.587+0000 7f2f99fe3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f94076b30 0x7f2f941137d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:03.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.587+0000 7f2f99fe3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f94076b30 0x7f2f941137d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51308/0 (socket says 172.21.15.59:51308) 2024-03-19T03:07:03.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.587+0000 7f2f99fe3700 1 -- 172.21.15.59:0/2629628687 learned_addr learned my addr 172.21.15.59:0/2629628687 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:03.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.587+0000 7f2f99fe3700 1 -- 172.21.15.59:0/2629628687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f900094f0 con 0x7f2f94076b30 2024-03-19T03:07:03.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.587+0000 7f2f99fe3700 1 --2- 172.21.15.59:0/2629628687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f94076b30 0x7f2f941137d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2f900090b0 tx=0x7f2f90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:03.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.587+0000 7f2f8a7fc700 1 -- 172.21.15.59:0/2629628687 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f90010760 con 0x7f2f94076b30 2024-03-19T03:07:03.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.587+0000 7f2f8a7fc700 1 -- 172.21.15.59:0/2629628687 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2f90010d50 con 0x7f2f94076b30 2024-03-19T03:07:03.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.587+0000 7f2f9c247700 1 -- 172.21.15.59:0/2629628687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f94113f10 con 0x7f2f94076b30 2024-03-19T03:07:03.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.587+0000 7f2f8a7fc700 1 -- 172.21.15.59:0/2629628687 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f9001ace0 con 0x7f2f94076b30 2024-03-19T03:07:03.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.587+0000 7f2f9c247700 1 -- 172.21.15.59:0/2629628687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f94114a80 con 0x7f2f94076b30 2024-03-19T03:07:03.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.588+0000 7f2f8a7fc700 1 -- 172.21.15.59:0/2629628687 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f2f9002d3f0 con 0x7f2f94076b30 2024-03-19T03:07:03.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.588+0000 7f2f8a7fc700 1 --2- 172.21.15.59:0/2629628687 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f2f80036790 0x7f2f80038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:03.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.588+0000 7f2f8a7fc700 1 -- 172.21.15.59:0/2629628687 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f2f9004a450 con 0x7f2f94076b30 2024-03-19T03:07:03.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.588+0000 7f2f997e2700 1 --2- 172.21.15.59:0/2629628687 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f2f80036790 0x7f2f80038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:03.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.588+0000 7f2f9c247700 1 -- 172.21.15.59:0/2629628687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f9405f5f0 con 0x7f2f94076b30 2024-03-19T03:07:03.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.589+0000 7f2f997e2700 1 --2- 172.21.15.59:0/2629628687 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f2f80036790 0x7f2f80038c40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2f84009990 tx=0x7f2f84006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:03.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.593+0000 7f2f8a7fc700 1 -- 172.21.15.59:0/2629628687 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f2f9002b8e0 con 0x7f2f94076b30 2024-03-19T03:07:03.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.800+0000 7f2f9c247700 1 -- 172.21.15.59:0/2629628687 --> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f940fe220 con 0x7f2f80036790 2024-03-19T03:07:03.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.805+0000 7f2f8a7fc700 1 -- 172.21.15.59:0/2629628687 <== mgr.14120 v2:172.21.15.59:6800/1573642037 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f2f940fe220 con 0x7f2f80036790 2024-03-19T03:07:03.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.807+0000 7f2f9c247700 1 -- 172.21.15.59:0/2629628687 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f2f80036790 msgr2=0x7f2f80038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:03.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.807+0000 7f2f9c247700 1 --2- 172.21.15.59:0/2629628687 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f2f80036790 0x7f2f80038c40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2f84009990 tx=0x7f2f84006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:03.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.807+0000 7f2f9c247700 1 -- 172.21.15.59:0/2629628687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f94076b30 msgr2=0x7f2f941137d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:03.040 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.807+0000 7f2f9c247700 1 --2- 172.21.15.59:0/2629628687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f94076b30 0x7f2f941137d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2f900090b0 tx=0x7f2f90009250 comp rx=0 tx=0).stop 2024-03-19T03:07:03.040 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.807+0000 7f2f9c247700 1 -- 172.21.15.59:0/2629628687 shutdown_connections 2024-03-19T03:07:03.040 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.807+0000 7f2f9c247700 1 --2- 172.21.15.59:0/2629628687 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f2f80036790 0x7f2f80038c40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:03.040 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.807+0000 7f2f9c247700 1 --2- 172.21.15.59:0/2629628687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f94076b30 0x7f2f941137d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:03.040 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.807+0000 7f2f9c247700 1 -- 172.21.15.59:0/2629628687 >> 172.21.15.59:0/2629628687 conn(0x7f2f940fce40 msgr2=0x7f2f940fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:03.040 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.807+0000 7f2f9c247700 1 -- 172.21.15.59:0/2629628687 shutdown_connections 2024-03-19T03:07:03.040 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:01.807+0000 7f2f9c247700 1 -- 172.21.15.59:0/2629628687 wait complete. 2024-03-19T03:07:03.040 INFO:teuthology.orchestra.run.smithi059.stdout:Deploying crash service with default placement... 2024-03-19T03:07:04.504 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:04 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:04.504 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:04 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:04.795 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-03-19T03:07:04.795 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.444+0000 7f10a4b09700 1 Processor -- start 2024-03-19T03:07:04.795 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.444+0000 7f10a4b09700 1 -- start start 2024-03-19T03:07:04.795 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.444+0000 7f10a4b09700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10a0102720 0x7f10a0102b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:04.795 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.444+0000 7f10a4b09700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a0103070 con 0x7f10a0102720 2024-03-19T03:07:04.796 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.444+0000 7f109e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10a0102720 0x7f10a0102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:04.796 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.444+0000 7f109e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10a0102720 0x7f10a0102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51310/0 (socket says 172.21.15.59:51310) 2024-03-19T03:07:04.796 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.444+0000 7f109e59c700 1 -- 172.21.15.59:0/694225457 learned_addr learned my addr 172.21.15.59:0/694225457 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:04.796 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.444+0000 7f109e59c700 1 -- 172.21.15.59:0/694225457 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a01031b0 con 0x7f10a0102720 2024-03-19T03:07:04.796 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.445+0000 7f109e59c700 1 --2- 172.21.15.59:0/694225457 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10a0102720 0x7f10a0102b30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1098009e50 tx=0x7f1098009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b3813de30781c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:04.796 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.445+0000 7f109d59a700 1 -- 172.21.15.59:0/694225457 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10980092e0 con 0x7f10a0102720 2024-03-19T03:07:04.796 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.445+0000 7f109d59a700 1 -- 172.21.15.59:0/694225457 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1098012430 con 0x7f10a0102720 2024-03-19T03:07:04.796 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.445+0000 7f109d59a700 1 -- 172.21.15.59:0/694225457 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10980075a0 con 0x7f10a0102720 2024-03-19T03:07:04.796 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.446+0000 7f10a4b09700 1 -- 172.21.15.59:0/694225457 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10a0102720 msgr2=0x7f10a0102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:04.796 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.446+0000 7f10a4b09700 1 --2- 172.21.15.59:0/694225457 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10a0102720 0x7f10a0102b30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1098009e50 tx=0x7f1098009840 comp rx=0 tx=0).stop 2024-03-19T03:07:04.796 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.446+0000 7f10a4b09700 1 -- 172.21.15.59:0/694225457 shutdown_connections 2024-03-19T03:07:04.796 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.446+0000 7f10a4b09700 1 --2- 172.21.15.59:0/694225457 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10a0102720 0x7f10a0102b30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:04.796 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.446+0000 7f10a4b09700 1 -- 172.21.15.59:0/694225457 >> 172.21.15.59:0/694225457 conn(0x7f10a00fde40 msgr2=0x7f10a01002b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:04.796 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.446+0000 7f10a4b09700 1 -- 172.21.15.59:0/694225457 shutdown_connections 2024-03-19T03:07:04.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.446+0000 7f10a4b09700 1 -- 172.21.15.59:0/694225457 wait complete. 2024-03-19T03:07:04.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.446+0000 7f10a4b09700 1 Processor -- start 2024-03-19T03:07:04.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.446+0000 7f10a4b09700 1 -- start start 2024-03-19T03:07:04.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.447+0000 7f10a4b09700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10a0102720 0x7f10a01115c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:04.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.447+0000 7f10a4b09700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a0111b00 con 0x7f10a0102720 2024-03-19T03:07:04.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.447+0000 7f109e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10a0102720 0x7f10a01115c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:04.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.447+0000 7f109e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10a0102720 0x7f10a01115c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51318/0 (socket says 172.21.15.59:51318) 2024-03-19T03:07:04.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.447+0000 7f109e59c700 1 -- 172.21.15.59:0/2769416389 learned_addr learned my addr 172.21.15.59:0/2769416389 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:04.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.447+0000 7f109e59c700 1 -- 172.21.15.59:0/2769416389 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1098009ae0 con 0x7f10a0102720 2024-03-19T03:07:04.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.447+0000 7f109e59c700 1 --2- 172.21.15.59:0/2769416389 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10a0102720 0x7f10a01115c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1098009fd0 tx=0x7f1098009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:04.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.447+0000 7f1096ffd700 1 -- 172.21.15.59:0/2769416389 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10980092e0 con 0x7f10a0102720 2024-03-19T03:07:04.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.447+0000 7f10a4b09700 1 -- 172.21.15.59:0/2769416389 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a0111d00 con 0x7f10a0102720 2024-03-19T03:07:04.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.447+0000 7f1096ffd700 1 -- 172.21.15.59:0/2769416389 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1098007ce0 con 0x7f10a0102720 2024-03-19T03:07:04.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.447+0000 7f1096ffd700 1 -- 172.21.15.59:0/2769416389 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1098025990 con 0x7f10a0102720 2024-03-19T03:07:04.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.447+0000 7f10a4b09700 1 -- 172.21.15.59:0/2769416389 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a0112870 con 0x7f10a0102720 2024-03-19T03:07:04.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.448+0000 7f1096ffd700 1 -- 172.21.15.59:0/2769416389 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f1098019050 con 0x7f10a0102720 2024-03-19T03:07:04.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.448+0000 7f10a4b09700 1 -- 172.21.15.59:0/2769416389 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a005f5f0 con 0x7f10a0102720 2024-03-19T03:07:04.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.449+0000 7f1096ffd700 1 --2- 172.21.15.59:0/2769416389 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f108c0367e0 0x7f108c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:04.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.449+0000 7f1096ffd700 1 -- 172.21.15.59:0/2769416389 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f109804a130 con 0x7f10a0102720 2024-03-19T03:07:04.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.449+0000 7f109dd9b700 1 --2- 172.21.15.59:0/2769416389 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f108c0367e0 0x7f108c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:04.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.449+0000 7f109dd9b700 1 --2- 172.21.15.59:0/2769416389 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f108c0367e0 0x7f108c038c90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1088009990 tx=0x7f1088006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:04.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.454+0000 7f1096ffd700 1 -- 172.21.15.59:0/2769416389 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f109802d9a0 con 0x7f10a0102720 2024-03-19T03:07:04.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.677+0000 7f10a4b09700 1 -- 172.21.15.59:0/2769416389 --> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f10a00ff180 con 0x7f108c0367e0 2024-03-19T03:07:04.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.682+0000 7f1096ffd700 1 -- 172.21.15.59:0/2769416389 <== mgr.14120 v2:172.21.15.59:6800/1573642037 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f10a00ff180 con 0x7f108c0367e0 2024-03-19T03:07:04.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.684+0000 7f10a4b09700 1 -- 172.21.15.59:0/2769416389 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f108c0367e0 msgr2=0x7f108c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:04.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.684+0000 7f10a4b09700 1 --2- 172.21.15.59:0/2769416389 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f108c0367e0 0x7f108c038c90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1088009990 tx=0x7f1088006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:04.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.684+0000 7f10a4b09700 1 -- 172.21.15.59:0/2769416389 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10a0102720 msgr2=0x7f10a01115c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:04.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.684+0000 7f10a4b09700 1 --2- 172.21.15.59:0/2769416389 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10a0102720 0x7f10a01115c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1098009fd0 tx=0x7f1098009840 comp rx=0 tx=0).stop 2024-03-19T03:07:04.799 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.684+0000 7f10a4b09700 1 -- 172.21.15.59:0/2769416389 shutdown_connections 2024-03-19T03:07:04.799 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.684+0000 7f10a4b09700 1 --2- 172.21.15.59:0/2769416389 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f108c0367e0 0x7f108c038c90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:04.799 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.684+0000 7f10a4b09700 1 --2- 172.21.15.59:0/2769416389 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10a0102720 0x7f10a01115c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:04.799 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.684+0000 7f10a4b09700 1 -- 172.21.15.59:0/2769416389 >> 172.21.15.59:0/2769416389 conn(0x7f10a00fde40 msgr2=0x7f10a00fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:04.799 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.684+0000 7f10a4b09700 1 -- 172.21.15.59:0/2769416389 shutdown_connections 2024-03-19T03:07:04.799 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:03.684+0000 7f10a4b09700 1 -- 172.21.15.59:0/2769416389 wait complete. 2024-03-19T03:07:04.799 INFO:teuthology.orchestra.run.smithi059.stdout:Deploying prometheus service with default placement... 2024-03-19T03:07:05.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:05 smithi059 ceph-mon[29133]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:07:05.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:05 smithi059 ceph-mon[29133]: Saving service crash spec with placement * 2024-03-19T03:07:05.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:05 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:05.969 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-03-19T03:07:05.969 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.220+0000 7f8d31abd700 1 Processor -- start 2024-03-19T03:07:05.969 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.220+0000 7f8d31abd700 1 -- start start 2024-03-19T03:07:05.969 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.220+0000 7f8d31abd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d2c1015d0 0x7f8d2c1019e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:05.969 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.220+0000 7f8d31abd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c101f20 con 0x7f8d2c1015d0 2024-03-19T03:07:05.969 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.221+0000 7f8d2b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d2c1015d0 0x7f8d2c1019e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:05.969 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.221+0000 7f8d2b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d2c1015d0 0x7f8d2c1019e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37420/0 (socket says 172.21.15.59:37420) 2024-03-19T03:07:05.969 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.221+0000 7f8d2b7fe700 1 -- 172.21.15.59:0/3333076368 learned_addr learned my addr 172.21.15.59:0/3333076368 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:05.969 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.221+0000 7f8d2b7fe700 1 -- 172.21.15.59:0/3333076368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c102060 con 0x7f8d2c1015d0 2024-03-19T03:07:05.969 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.221+0000 7f8d2b7fe700 1 --2- 172.21.15.59:0/3333076368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d2c1015d0 0x7f8d2c1019e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c0098e0 tx=0x7f8d1c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f1263baa45a490d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:05.969 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.221+0000 7f8d2a7fc700 1 -- 172.21.15.59:0/3333076368 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d1c00b860 con 0x7f8d2c1015d0 2024-03-19T03:07:05.969 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.221+0000 7f8d2a7fc700 1 -- 172.21.15.59:0/3333076368 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d1c00be90 con 0x7f8d2c1015d0 2024-03-19T03:07:05.970 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.221+0000 7f8d2a7fc700 1 -- 172.21.15.59:0/3333076368 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d1c0103c0 con 0x7f8d2c1015d0 2024-03-19T03:07:05.970 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.222+0000 7f8d31abd700 1 -- 172.21.15.59:0/3333076368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d2c1015d0 msgr2=0x7f8d2c1019e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:05.970 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.222+0000 7f8d31abd700 1 --2- 172.21.15.59:0/3333076368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d2c1015d0 0x7f8d2c1019e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c0098e0 tx=0x7f8d1c009250 comp rx=0 tx=0).stop 2024-03-19T03:07:05.970 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.222+0000 7f8d31abd700 1 -- 172.21.15.59:0/3333076368 shutdown_connections 2024-03-19T03:07:05.970 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.222+0000 7f8d31abd700 1 --2- 172.21.15.59:0/3333076368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d2c1015d0 0x7f8d2c1019e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:05.970 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.222+0000 7f8d31abd700 1 -- 172.21.15.59:0/3333076368 >> 172.21.15.59:0/3333076368 conn(0x7f8d2c0fcd50 msgr2=0x7f8d2c0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:05.970 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.222+0000 7f8d31abd700 1 -- 172.21.15.59:0/3333076368 shutdown_connections 2024-03-19T03:07:05.970 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.222+0000 7f8d31abd700 1 -- 172.21.15.59:0/3333076368 wait complete. 2024-03-19T03:07:05.970 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.223+0000 7f8d31abd700 1 Processor -- start 2024-03-19T03:07:05.970 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.223+0000 7f8d31abd700 1 -- start start 2024-03-19T03:07:05.970 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.223+0000 7f8d31abd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d2c1015d0 0x7f8d2c071200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:05.970 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.223+0000 7f8d31abd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c112110 con 0x7f8d2c1015d0 2024-03-19T03:07:05.970 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.223+0000 7f8d2b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d2c1015d0 0x7f8d2c071200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:05.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.223+0000 7f8d2b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d2c1015d0 0x7f8d2c071200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37436/0 (socket says 172.21.15.59:37436) 2024-03-19T03:07:05.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.223+0000 7f8d2b7fe700 1 -- 172.21.15.59:0/3980588536 learned_addr learned my addr 172.21.15.59:0/3980588536 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:05.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.223+0000 7f8d2b7fe700 1 -- 172.21.15.59:0/3980588536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1c0094f0 con 0x7f8d2c1015d0 2024-03-19T03:07:05.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.224+0000 7f8d2b7fe700 1 --2- 172.21.15.59:0/3980588536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d2c1015d0 0x7f8d2c071200 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c011c10 tx=0x7f8d1c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:05.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.224+0000 7f8d13fff700 1 -- 172.21.15.59:0/3980588536 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d1c0118f0 con 0x7f8d2c1015d0 2024-03-19T03:07:05.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.224+0000 7f8d31abd700 1 -- 172.21.15.59:0/3980588536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2c116720 con 0x7f8d2c1015d0 2024-03-19T03:07:05.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.224+0000 7f8d13fff700 1 -- 172.21.15.59:0/3980588536 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d1c010d60 con 0x7f8d2c1015d0 2024-03-19T03:07:05.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.224+0000 7f8d13fff700 1 -- 172.21.15.59:0/3980588536 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d1c0243f0 con 0x7f8d2c1015d0 2024-03-19T03:07:05.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.224+0000 7f8d31abd700 1 -- 172.21.15.59:0/3980588536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2c116af0 con 0x7f8d2c1015d0 2024-03-19T03:07:05.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.225+0000 7f8d13fff700 1 -- 172.21.15.59:0/3980588536 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f8d1c02d3f0 con 0x7f8d2c1015d0 2024-03-19T03:07:05.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.225+0000 7f8d31abd700 1 -- 172.21.15.59:0/3980588536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2c04bed0 con 0x7f8d2c1015d0 2024-03-19T03:07:05.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.225+0000 7f8d13fff700 1 --2- 172.21.15.59:0/3980588536 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f8d14036790 0x7f8d14038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:05.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.225+0000 7f8d13fff700 1 -- 172.21.15.59:0/3980588536 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f8d1c04a200 con 0x7f8d2c1015d0 2024-03-19T03:07:05.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.225+0000 7f8d2affd700 1 --2- 172.21.15.59:0/3980588536 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f8d14036790 0x7f8d14038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:05.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.226+0000 7f8d2affd700 1 --2- 172.21.15.59:0/3980588536 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f8d14036790 0x7f8d14038c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8d20009990 tx=0x7f8d20006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:05.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.230+0000 7f8d13fff700 1 -- 172.21.15.59:0/3980588536 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f8d1c02b8e0 con 0x7f8d2c1015d0 2024-03-19T03:07:05.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.441+0000 7f8d31abd700 1 -- 172.21.15.59:0/3980588536 --> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d2c0fe0a0 con 0x7f8d14036790 2024-03-19T03:07:05.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.446+0000 7f8d13fff700 1 -- 172.21.15.59:0/3980588536 <== mgr.14120 v2:172.21.15.59:6800/1573642037 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f8d2c0fe0a0 con 0x7f8d14036790 2024-03-19T03:07:05.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.448+0000 7f8d31abd700 1 -- 172.21.15.59:0/3980588536 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f8d14036790 msgr2=0x7f8d14038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:05.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.448+0000 7f8d31abd700 1 --2- 172.21.15.59:0/3980588536 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f8d14036790 0x7f8d14038c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8d20009990 tx=0x7f8d20006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:05.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.448+0000 7f8d31abd700 1 -- 172.21.15.59:0/3980588536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d2c1015d0 msgr2=0x7f8d2c071200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:05.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.448+0000 7f8d31abd700 1 --2- 172.21.15.59:0/3980588536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d2c1015d0 0x7f8d2c071200 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c011c10 tx=0x7f8d1c009250 comp rx=0 tx=0).stop 2024-03-19T03:07:05.973 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.448+0000 7f8d31abd700 1 -- 172.21.15.59:0/3980588536 shutdown_connections 2024-03-19T03:07:05.973 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.448+0000 7f8d31abd700 1 --2- 172.21.15.59:0/3980588536 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f8d14036790 0x7f8d14038c40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:05.973 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.448+0000 7f8d31abd700 1 --2- 172.21.15.59:0/3980588536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d2c1015d0 0x7f8d2c071200 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:05.973 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.448+0000 7f8d31abd700 1 -- 172.21.15.59:0/3980588536 >> 172.21.15.59:0/3980588536 conn(0x7f8d2c0fcd50 msgr2=0x7f8d2c0fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:05.973 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.448+0000 7f8d31abd700 1 -- 172.21.15.59:0/3980588536 shutdown_connections 2024-03-19T03:07:05.973 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:05.449+0000 7f8d31abd700 1 -- 172.21.15.59:0/3980588536 wait complete. 2024-03-19T03:07:05.973 INFO:teuthology.orchestra.run.smithi059.stdout:Deploying grafana service with default placement... 2024-03-19T03:07:06.465 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:06 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:06.465 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:06 smithi059 ceph-mon[29133]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:07:06.465 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:06 smithi059 ceph-mon[29133]: Saving service prometheus spec with placement count:1 2024-03-19T03:07:06.465 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:06 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:07.815 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:07 smithi059 ceph-mon[29133]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:07:07.815 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:07 smithi059 ceph-mon[29133]: Saving service grafana spec with placement count:1 2024-03-19T03:07:07.815 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:07 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:07.823 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-03-19T03:07:07.823 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.405+0000 7f0376530700 1 Processor -- start 2024-03-19T03:07:07.823 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.405+0000 7f0376530700 1 -- start start 2024-03-19T03:07:07.823 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.405+0000 7f0376530700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03701015b0 0x7f03701019c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:07.823 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.405+0000 7f0376530700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0370101f00 con 0x7f03701015b0 2024-03-19T03:07:07.824 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.405+0000 7f036ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03701015b0 0x7f03701019c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:07.824 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.405+0000 7f036ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03701015b0 0x7f03701019c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37444/0 (socket says 172.21.15.59:37444) 2024-03-19T03:07:07.824 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.405+0000 7f036ffff700 1 -- 172.21.15.59:0/4233044923 learned_addr learned my addr 172.21.15.59:0/4233044923 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:07.824 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.405+0000 7f036ffff700 1 -- 172.21.15.59:0/4233044923 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0370102040 con 0x7f03701015b0 2024-03-19T03:07:07.824 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.405+0000 7f036ffff700 1 --2- 172.21.15.59:0/4233044923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03701015b0 0x7f03701019c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f036000bb80 tx=0x7f036000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32128dc577e9c240 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:07.824 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.406+0000 7f036effd700 1 -- 172.21.15.59:0/4233044923 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f036000dd00 con 0x7f03701015b0 2024-03-19T03:07:07.824 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.406+0000 7f036effd700 1 -- 172.21.15.59:0/4233044923 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f036000de60 con 0x7f03701015b0 2024-03-19T03:07:07.824 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.406+0000 7f036effd700 1 -- 172.21.15.59:0/4233044923 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0360018920 con 0x7f03701015b0 2024-03-19T03:07:07.824 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.406+0000 7f0376530700 1 -- 172.21.15.59:0/4233044923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03701015b0 msgr2=0x7f03701019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:07.824 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.406+0000 7f0376530700 1 --2- 172.21.15.59:0/4233044923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03701015b0 0x7f03701019c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f036000bb80 tx=0x7f036000b4f0 comp rx=0 tx=0).stop 2024-03-19T03:07:07.824 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.406+0000 7f0376530700 1 -- 172.21.15.59:0/4233044923 shutdown_connections 2024-03-19T03:07:07.824 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.406+0000 7f0376530700 1 --2- 172.21.15.59:0/4233044923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03701015b0 0x7f03701019c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:07.824 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.406+0000 7f0376530700 1 -- 172.21.15.59:0/4233044923 >> 172.21.15.59:0/4233044923 conn(0x7f03700fcd50 msgr2=0x7f03700ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:07.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.406+0000 7f0376530700 1 -- 172.21.15.59:0/4233044923 shutdown_connections 2024-03-19T03:07:07.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.406+0000 7f0376530700 1 -- 172.21.15.59:0/4233044923 wait complete. 2024-03-19T03:07:07.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.407+0000 7f0376530700 1 Processor -- start 2024-03-19T03:07:07.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.407+0000 7f0376530700 1 -- start start 2024-03-19T03:07:07.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.407+0000 7f0376530700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03701015b0 0x7f037010f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:07.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.407+0000 7f0376530700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037010f870 con 0x7f03701015b0 2024-03-19T03:07:07.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.407+0000 7f036ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03701015b0 0x7f037010f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:07.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.407+0000 7f036ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03701015b0 0x7f037010f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37454/0 (socket says 172.21.15.59:37454) 2024-03-19T03:07:07.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.407+0000 7f036ffff700 1 -- 172.21.15.59:0/724238063 learned_addr learned my addr 172.21.15.59:0/724238063 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:07.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.407+0000 7f036ffff700 1 -- 172.21.15.59:0/724238063 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036000b790 con 0x7f03701015b0 2024-03-19T03:07:07.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.408+0000 7f036ffff700 1 --2- 172.21.15.59:0/724238063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03701015b0 0x7f037010f330 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f03600106a0 tx=0x7f036000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:07.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.408+0000 7f036cff9700 1 -- 172.21.15.59:0/724238063 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f036000dc70 con 0x7f03701015b0 2024-03-19T03:07:07.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.408+0000 7f036cff9700 1 -- 172.21.15.59:0/724238063 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0360010b80 con 0x7f03701015b0 2024-03-19T03:07:07.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.408+0000 7f036cff9700 1 -- 172.21.15.59:0/724238063 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0360021550 con 0x7f03701015b0 2024-03-19T03:07:07.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.408+0000 7f0376530700 1 -- 172.21.15.59:0/724238063 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f037010fa70 con 0x7f03701015b0 2024-03-19T03:07:07.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.408+0000 7f0376530700 1 -- 172.21.15.59:0/724238063 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f037010ff00 con 0x7f03701015b0 2024-03-19T03:07:07.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.409+0000 7f036cff9700 1 -- 172.21.15.59:0/724238063 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f03600217c0 con 0x7f03701015b0 2024-03-19T03:07:07.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.409+0000 7f036cff9700 1 --2- 172.21.15.59:0/724238063 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f03580367e0 0x7f0358038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:07.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.409+0000 7f0376530700 1 -- 172.21.15.59:0/724238063 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f037004bed0 con 0x7f03701015b0 2024-03-19T03:07:07.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.409+0000 7f036cff9700 1 -- 172.21.15.59:0/724238063 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f0360014070 con 0x7f03701015b0 2024-03-19T03:07:07.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.409+0000 7f036f7fe700 1 --2- 172.21.15.59:0/724238063 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f03580367e0 0x7f0358038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:07.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.409+0000 7f036f7fe700 1 --2- 172.21.15.59:0/724238063 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f03580367e0 0x7f0358038c90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0364009990 tx=0x7f0364006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:07.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.414+0000 7f036cff9700 1 -- 172.21.15.59:0/724238063 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f036000edc0 con 0x7f03701015b0 2024-03-19T03:07:07.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.620+0000 7f0376530700 1 -- 172.21.15.59:0/724238063 --> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f0370110aa0 con 0x7f03580367e0 2024-03-19T03:07:07.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.624+0000 7f036cff9700 1 -- 172.21.15.59:0/724238063 <== mgr.14120 v2:172.21.15.59:6800/1573642037 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f0370110aa0 con 0x7f03580367e0 2024-03-19T03:07:07.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.626+0000 7f0376530700 1 -- 172.21.15.59:0/724238063 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f03580367e0 msgr2=0x7f0358038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:07.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.626+0000 7f0376530700 1 --2- 172.21.15.59:0/724238063 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f03580367e0 0x7f0358038c90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0364009990 tx=0x7f0364006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:07.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.626+0000 7f0376530700 1 -- 172.21.15.59:0/724238063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03701015b0 msgr2=0x7f037010f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:07.827 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.626+0000 7f0376530700 1 --2- 172.21.15.59:0/724238063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03701015b0 0x7f037010f330 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f03600106a0 tx=0x7f036000b4f0 comp rx=0 tx=0).stop 2024-03-19T03:07:07.827 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.626+0000 7f0376530700 1 -- 172.21.15.59:0/724238063 shutdown_connections 2024-03-19T03:07:07.827 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.626+0000 7f0376530700 1 --2- 172.21.15.59:0/724238063 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f03580367e0 0x7f0358038c90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:07.827 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.626+0000 7f0376530700 1 --2- 172.21.15.59:0/724238063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03701015b0 0x7f037010f330 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:07.827 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.626+0000 7f0376530700 1 -- 172.21.15.59:0/724238063 >> 172.21.15.59:0/724238063 conn(0x7f03700fcd50 msgr2=0x7f0370105b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:07.827 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.626+0000 7f0376530700 1 -- 172.21.15.59:0/724238063 shutdown_connections 2024-03-19T03:07:07.827 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:06.626+0000 7f0376530700 1 -- 172.21.15.59:0/724238063 wait complete. 2024-03-19T03:07:07.827 INFO:teuthology.orchestra.run.smithi059.stdout:Deploying node-exporter service with default placement... 2024-03-19T03:07:09.146 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-03-19T03:07:09.147 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.224+0000 7f8399126700 1 Processor -- start 2024-03-19T03:07:09.147 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.224+0000 7f8399126700 1 -- start start 2024-03-19T03:07:09.147 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.224+0000 7f8399126700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8394078c80 0x7f83940771e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:09.147 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.224+0000 7f8399126700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8394079090 con 0x7f8394078c80 2024-03-19T03:07:09.147 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.225+0000 7f8392d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8394078c80 0x7f83940771e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:09.147 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.225+0000 7f8392d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8394078c80 0x7f83940771e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37468/0 (socket says 172.21.15.59:37468) 2024-03-19T03:07:09.147 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.225+0000 7f8392d9d700 1 -- 172.21.15.59:0/3237474561 learned_addr learned my addr 172.21.15.59:0/3237474561 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:09.147 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.225+0000 7f8392d9d700 1 -- 172.21.15.59:0/3237474561 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8394077720 con 0x7f8394078c80 2024-03-19T03:07:09.147 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.225+0000 7f8392d9d700 1 --2- 172.21.15.59:0/3237474561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8394078c80 0x7f83940771e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f83880098e0 tx=0x7f8388009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fecd7f9ccfb811e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:09.147 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.225+0000 7f8391d9b700 1 -- 172.21.15.59:0/3237474561 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f838800b860 con 0x7f8394078c80 2024-03-19T03:07:09.147 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.225+0000 7f8391d9b700 1 -- 172.21.15.59:0/3237474561 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f838800be90 con 0x7f8394078c80 2024-03-19T03:07:09.147 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.225+0000 7f8391d9b700 1 -- 172.21.15.59:0/3237474561 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83880103c0 con 0x7f8394078c80 2024-03-19T03:07:09.147 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.226+0000 7f8399126700 1 -- 172.21.15.59:0/3237474561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8394078c80 msgr2=0x7f83940771e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:09.147 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.226+0000 7f8399126700 1 --2- 172.21.15.59:0/3237474561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8394078c80 0x7f83940771e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f83880098e0 tx=0x7f8388009250 comp rx=0 tx=0).stop 2024-03-19T03:07:09.148 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.226+0000 7f8399126700 1 -- 172.21.15.59:0/3237474561 shutdown_connections 2024-03-19T03:07:09.148 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.226+0000 7f8399126700 1 --2- 172.21.15.59:0/3237474561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8394078c80 0x7f83940771e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:09.148 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.226+0000 7f8399126700 1 -- 172.21.15.59:0/3237474561 >> 172.21.15.59:0/3237474561 conn(0x7f83940fdee0 msgr2=0x7f83941002f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:09.148 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.226+0000 7f8399126700 1 -- 172.21.15.59:0/3237474561 shutdown_connections 2024-03-19T03:07:09.148 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.226+0000 7f8399126700 1 -- 172.21.15.59:0/3237474561 wait complete. 2024-03-19T03:07:09.148 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.226+0000 7f8399126700 1 Processor -- start 2024-03-19T03:07:09.148 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.227+0000 7f8399126700 1 -- start start 2024-03-19T03:07:09.148 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.227+0000 7f8399126700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8394078c80 0x7f8394113810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:09.148 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.227+0000 7f8399126700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8394113d50 con 0x7f8394078c80 2024-03-19T03:07:09.148 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.227+0000 7f8392d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8394078c80 0x7f8394113810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:09.148 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.227+0000 7f8392d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8394078c80 0x7f8394113810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37480/0 (socket says 172.21.15.59:37480) 2024-03-19T03:07:09.148 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.227+0000 7f8392d9d700 1 -- 172.21.15.59:0/3115934800 learned_addr learned my addr 172.21.15.59:0/3115934800 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:09.148 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.227+0000 7f8392d9d700 1 -- 172.21.15.59:0/3115934800 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83880094f0 con 0x7f8394078c80 2024-03-19T03:07:09.149 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.227+0000 7f8392d9d700 1 --2- 172.21.15.59:0/3115934800 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8394078c80 0x7f8394113810 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f83880094c0 tx=0x7f8388009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:09.149 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.227+0000 7f837b7fe700 1 -- 172.21.15.59:0/3115934800 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8388010680 con 0x7f8394078c80 2024-03-19T03:07:09.149 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.228+0000 7f837b7fe700 1 -- 172.21.15.59:0/3115934800 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8388010cb0 con 0x7f8394078c80 2024-03-19T03:07:09.149 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.228+0000 7f8399126700 1 -- 172.21.15.59:0/3115934800 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8394113f50 con 0x7f8394078c80 2024-03-19T03:07:09.149 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.228+0000 7f837b7fe700 1 -- 172.21.15.59:0/3115934800 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f838801ad40 con 0x7f8394078c80 2024-03-19T03:07:09.149 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.228+0000 7f8399126700 1 -- 172.21.15.59:0/3115934800 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8394114ac0 con 0x7f8394078c80 2024-03-19T03:07:09.149 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.228+0000 7f837b7fe700 1 -- 172.21.15.59:0/3115934800 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f8388025400 con 0x7f8394078c80 2024-03-19T03:07:09.149 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.229+0000 7f837b7fe700 1 --2- 172.21.15.59:0/3115934800 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f837c036790 0x7f837c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:09.149 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.229+0000 7f8399126700 1 -- 172.21.15.59:0/3115934800 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f839405f5f0 con 0x7f8394078c80 2024-03-19T03:07:09.149 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.229+0000 7f837b7fe700 1 -- 172.21.15.59:0/3115934800 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f838804a2e0 con 0x7f8394078c80 2024-03-19T03:07:09.149 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.229+0000 7f839259c700 1 --2- 172.21.15.59:0/3115934800 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f837c036790 0x7f837c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:09.149 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.229+0000 7f839259c700 1 --2- 172.21.15.59:0/3115934800 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f837c036790 0x7f837c038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8384009990 tx=0x7f8384006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:09.150 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.233+0000 7f837b7fe700 1 -- 172.21.15.59:0/3115934800 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f838802c8f0 con 0x7f8394078c80 2024-03-19T03:07:09.150 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.435+0000 7f8399126700 1 -- 172.21.15.59:0/3115934800 --> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f83940ff260 con 0x7f837c036790 2024-03-19T03:07:09.150 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.440+0000 7f837b7fe700 1 -- 172.21.15.59:0/3115934800 <== mgr.14120 v2:172.21.15.59:6800/1573642037 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f83940ff260 con 0x7f837c036790 2024-03-19T03:07:09.150 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.442+0000 7f8399126700 1 -- 172.21.15.59:0/3115934800 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f837c036790 msgr2=0x7f837c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:09.150 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.442+0000 7f8399126700 1 --2- 172.21.15.59:0/3115934800 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f837c036790 0x7f837c038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8384009990 tx=0x7f8384006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:09.150 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.442+0000 7f8399126700 1 -- 172.21.15.59:0/3115934800 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8394078c80 msgr2=0x7f8394113810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:09.150 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.442+0000 7f8399126700 1 --2- 172.21.15.59:0/3115934800 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8394078c80 0x7f8394113810 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f83880094c0 tx=0x7f8388009250 comp rx=0 tx=0).stop 2024-03-19T03:07:09.150 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.442+0000 7f8399126700 1 -- 172.21.15.59:0/3115934800 shutdown_connections 2024-03-19T03:07:09.150 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.442+0000 7f8399126700 1 --2- 172.21.15.59:0/3115934800 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f837c036790 0x7f837c038c40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:09.150 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.442+0000 7f8399126700 1 --2- 172.21.15.59:0/3115934800 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8394078c80 0x7f8394113810 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:09.150 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.442+0000 7f8399126700 1 -- 172.21.15.59:0/3115934800 >> 172.21.15.59:0/3115934800 conn(0x7f83940fdee0 msgr2=0x7f83940fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:09.150 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.442+0000 7f8399126700 1 -- 172.21.15.59:0/3115934800 shutdown_connections 2024-03-19T03:07:09.150 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:08.442+0000 7f8399126700 1 -- 172.21.15.59:0/3115934800 wait complete. 2024-03-19T03:07:09.151 INFO:teuthology.orchestra.run.smithi059.stdout:Deploying alertmanager service with default placement... 2024-03-19T03:07:09.574 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:09 smithi059 ceph-mon[29133]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:07:09.574 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:09 smithi059 ceph-mon[29133]: Saving service node-exporter spec with placement * 2024-03-19T03:07:09.574 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:09 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:10.377 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-03-19T03:07:10.378 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.606+0000 7f70ecf7a700 1 Processor -- start 2024-03-19T03:07:10.378 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.606+0000 7f70ecf7a700 1 -- start start 2024-03-19T03:07:10.378 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.606+0000 7f70ecf7a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70e8103d20 0x7f70e8104130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:10.378 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.606+0000 7f70ecf7a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e8104670 con 0x7f70e8103d20 2024-03-19T03:07:10.378 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.606+0000 7f70e659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70e8103d20 0x7f70e8104130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:10.378 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.606+0000 7f70e659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70e8103d20 0x7f70e8104130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37488/0 (socket says 172.21.15.59:37488) 2024-03-19T03:07:10.378 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.606+0000 7f70e659c700 1 -- 172.21.15.59:0/549017536 learned_addr learned my addr 172.21.15.59:0/549017536 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:10.378 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.607+0000 7f70e659c700 1 -- 172.21.15.59:0/549017536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e81047b0 con 0x7f70e8103d20 2024-03-19T03:07:10.378 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.607+0000 7f70e659c700 1 --2- 172.21.15.59:0/549017536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70e8103d20 0x7f70e8104130 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f70d80098e0 tx=0x7f70d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d91a9cd5600005b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:10.378 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.607+0000 7f70e559a700 1 -- 172.21.15.59:0/549017536 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70d800b860 con 0x7f70e8103d20 2024-03-19T03:07:10.378 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.607+0000 7f70e559a700 1 -- 172.21.15.59:0/549017536 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f70d800be90 con 0x7f70e8103d20 2024-03-19T03:07:10.379 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.607+0000 7f70e559a700 1 -- 172.21.15.59:0/549017536 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70d80103c0 con 0x7f70e8103d20 2024-03-19T03:07:10.379 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.608+0000 7f70ecf7a700 1 -- 172.21.15.59:0/549017536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70e8103d20 msgr2=0x7f70e8104130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:10.379 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.608+0000 7f70ecf7a700 1 --2- 172.21.15.59:0/549017536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70e8103d20 0x7f70e8104130 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f70d80098e0 tx=0x7f70d8009250 comp rx=0 tx=0).stop 2024-03-19T03:07:10.379 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.608+0000 7f70ecf7a700 1 -- 172.21.15.59:0/549017536 shutdown_connections 2024-03-19T03:07:10.379 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.608+0000 7f70ecf7a700 1 --2- 172.21.15.59:0/549017536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70e8103d20 0x7f70e8104130 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:10.379 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.608+0000 7f70ecf7a700 1 -- 172.21.15.59:0/549017536 >> 172.21.15.59:0/549017536 conn(0x7f70e80fc500 msgr2=0x7f70e80fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:10.379 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.608+0000 7f70ecf7a700 1 -- 172.21.15.59:0/549017536 shutdown_connections 2024-03-19T03:07:10.379 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.608+0000 7f70ecf7a700 1 -- 172.21.15.59:0/549017536 wait complete. 2024-03-19T03:07:10.379 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.608+0000 7f70ecf7a700 1 Processor -- start 2024-03-19T03:07:10.379 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.608+0000 7f70ecf7a700 1 -- start start 2024-03-19T03:07:10.379 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.609+0000 7f70ecf7a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70e8103d20 0x7f70e8111590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:10.379 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.609+0000 7f70ecf7a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e8111ad0 con 0x7f70e8103d20 2024-03-19T03:07:10.379 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.609+0000 7f70e659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70e8103d20 0x7f70e8111590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:10.379 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.609+0000 7f70e659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70e8103d20 0x7f70e8111590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37498/0 (socket says 172.21.15.59:37498) 2024-03-19T03:07:10.379 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.609+0000 7f70e659c700 1 -- 172.21.15.59:0/1816606720 learned_addr learned my addr 172.21.15.59:0/1816606720 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:10.380 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.609+0000 7f70e659c700 1 -- 172.21.15.59:0/1816606720 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d80094f0 con 0x7f70e8103d20 2024-03-19T03:07:10.380 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.609+0000 7f70e659c700 1 --2- 172.21.15.59:0/1816606720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70e8103d20 0x7f70e8111590 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f70d8011680 tx=0x7f70d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:10.380 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.609+0000 7f70d6ffd700 1 -- 172.21.15.59:0/1816606720 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70d8010890 con 0x7f70e8103d20 2024-03-19T03:07:10.380 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.609+0000 7f70ecf7a700 1 -- 172.21.15.59:0/1816606720 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e8111cd0 con 0x7f70e8103d20 2024-03-19T03:07:10.380 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.609+0000 7f70d6ffd700 1 -- 172.21.15.59:0/1816606720 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f70d8010ec0 con 0x7f70e8103d20 2024-03-19T03:07:10.380 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.609+0000 7f70d6ffd700 1 -- 172.21.15.59:0/1816606720 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70d80243f0 con 0x7f70e8103d20 2024-03-19T03:07:10.380 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.609+0000 7f70ecf7a700 1 -- 172.21.15.59:0/1816606720 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e8112840 con 0x7f70e8103d20 2024-03-19T03:07:10.380 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.610+0000 7f70d6ffd700 1 -- 172.21.15.59:0/1816606720 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f70d80109f0 con 0x7f70e8103d20 2024-03-19T03:07:10.381 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.610+0000 7f70ecf7a700 1 -- 172.21.15.59:0/1816606720 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e805f5f0 con 0x7f70e8103d20 2024-03-19T03:07:10.381 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.610+0000 7f70d6ffd700 1 --2- 172.21.15.59:0/1816606720 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f70d0036790 0x7f70d0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:10.381 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.610+0000 7f70d6ffd700 1 -- 172.21.15.59:0/1816606720 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f70d804a270 con 0x7f70e8103d20 2024-03-19T03:07:10.381 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.610+0000 7f70e5d9b700 1 --2- 172.21.15.59:0/1816606720 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f70d0036790 0x7f70d0038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:10.382 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.611+0000 7f70e5d9b700 1 --2- 172.21.15.59:0/1816606720 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f70d0036790 0x7f70d0038c40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f70dc009990 tx=0x7f70dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:10.382 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.615+0000 7f70d6ffd700 1 -- 172.21.15.59:0/1816606720 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f70d802b4d0 con 0x7f70e8103d20 2024-03-19T03:07:10.382 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.817+0000 7f70ecf7a700 1 -- 172.21.15.59:0/1816606720 --> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f70e80fd7b0 con 0x7f70d0036790 2024-03-19T03:07:10.382 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.823+0000 7f70d6ffd700 1 -- 172.21.15.59:0/1816606720 <== mgr.14120 v2:172.21.15.59:6800/1573642037 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f70e80fd7b0 con 0x7f70d0036790 2024-03-19T03:07:10.382 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.825+0000 7f70ecf7a700 1 -- 172.21.15.59:0/1816606720 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f70d0036790 msgr2=0x7f70d0038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:10.382 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.825+0000 7f70ecf7a700 1 --2- 172.21.15.59:0/1816606720 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f70d0036790 0x7f70d0038c40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f70dc009990 tx=0x7f70dc006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:10.382 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.825+0000 7f70ecf7a700 1 -- 172.21.15.59:0/1816606720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70e8103d20 msgr2=0x7f70e8111590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:10.382 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.825+0000 7f70ecf7a700 1 --2- 172.21.15.59:0/1816606720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70e8103d20 0x7f70e8111590 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f70d8011680 tx=0x7f70d8009250 comp rx=0 tx=0).stop 2024-03-19T03:07:10.382 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.825+0000 7f70ecf7a700 1 -- 172.21.15.59:0/1816606720 shutdown_connections 2024-03-19T03:07:10.382 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.825+0000 7f70ecf7a700 1 --2- 172.21.15.59:0/1816606720 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f70d0036790 0x7f70d0038c40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:10.382 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.825+0000 7f70ecf7a700 1 --2- 172.21.15.59:0/1816606720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70e8103d20 0x7f70e8111590 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:10.382 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.825+0000 7f70ecf7a700 1 -- 172.21.15.59:0/1816606720 >> 172.21.15.59:0/1816606720 conn(0x7f70e80fc500 msgr2=0x7f70e80fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:10.382 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.825+0000 7f70ecf7a700 1 -- 172.21.15.59:0/1816606720 shutdown_connections 2024-03-19T03:07:10.383 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:09.825+0000 7f70ecf7a700 1 -- 172.21.15.59:0/1816606720 wait complete. 2024-03-19T03:07:11.008 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:10 smithi059 ceph-mon[29133]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:07:11.008 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:10 smithi059 ceph-mon[29133]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:07:11.008 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:10 smithi059 ceph-mon[29133]: Saving service alertmanager spec with placement count:1 2024-03-19T03:07:11.008 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:10 smithi059 ceph-mon[29133]: from='mgr.14120 172.21.15.59:0/1073659959' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:11.601 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.807+0000 7effb9917700 1 Processor -- start 2024-03-19T03:07:11.601 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.807+0000 7effb9917700 1 -- start start 2024-03-19T03:07:11.601 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.807+0000 7effb9917700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effb4102bb0 0x7effb4102fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:11.601 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.807+0000 7effb9917700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb4103500 con 0x7effb4102bb0 2024-03-19T03:07:11.601 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.808+0000 7effb2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effb4102bb0 0x7effb4102fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:11.601 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.808+0000 7effb2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effb4102bb0 0x7effb4102fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37508/0 (socket says 172.21.15.59:37508) 2024-03-19T03:07:11.601 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.808+0000 7effb2ffd700 1 -- 172.21.15.59:0/2321402883 learned_addr learned my addr 172.21.15.59:0/2321402883 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:11.601 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.808+0000 7effb2ffd700 1 -- 172.21.15.59:0/2321402883 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb4103640 con 0x7effb4102bb0 2024-03-19T03:07:11.601 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.808+0000 7effb2ffd700 1 --2- 172.21.15.59:0/2321402883 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effb4102bb0 0x7effb4102fc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7effac0098e0 tx=0x7effac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb01f70c2fffee31 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:11.601 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.808+0000 7effb1ffb700 1 -- 172.21.15.59:0/2321402883 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effac00b860 con 0x7effb4102bb0 2024-03-19T03:07:11.601 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.808+0000 7effb1ffb700 1 -- 172.21.15.59:0/2321402883 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effac00be90 con 0x7effb4102bb0 2024-03-19T03:07:11.601 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.808+0000 7effb1ffb700 1 -- 172.21.15.59:0/2321402883 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effac0103c0 con 0x7effb4102bb0 2024-03-19T03:07:11.602 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.809+0000 7effb9917700 1 -- 172.21.15.59:0/2321402883 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effb4102bb0 msgr2=0x7effb4102fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:11.602 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.809+0000 7effb9917700 1 --2- 172.21.15.59:0/2321402883 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effb4102bb0 0x7effb4102fc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7effac0098e0 tx=0x7effac009250 comp rx=0 tx=0).stop 2024-03-19T03:07:11.602 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.809+0000 7effb9917700 1 -- 172.21.15.59:0/2321402883 shutdown_connections 2024-03-19T03:07:11.602 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.809+0000 7effb9917700 1 --2- 172.21.15.59:0/2321402883 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effb4102bb0 0x7effb4102fc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:11.602 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.809+0000 7effb9917700 1 -- 172.21.15.59:0/2321402883 >> 172.21.15.59:0/2321402883 conn(0x7effb40fc500 msgr2=0x7effb40fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:11.602 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.809+0000 7effb9917700 1 -- 172.21.15.59:0/2321402883 shutdown_connections 2024-03-19T03:07:11.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.809+0000 7effb9917700 1 -- 172.21.15.59:0/2321402883 wait complete. 2024-03-19T03:07:11.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.810+0000 7effb9917700 1 Processor -- start 2024-03-19T03:07:11.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.810+0000 7effb9917700 1 -- start start 2024-03-19T03:07:11.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.810+0000 7effb9917700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effb4102bb0 0x7effb4107880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:11.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.810+0000 7effb9917700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb4107dc0 con 0x7effb4102bb0 2024-03-19T03:07:11.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.810+0000 7effb2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effb4102bb0 0x7effb4107880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:11.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.810+0000 7effb2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effb4102bb0 0x7effb4107880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37522/0 (socket says 172.21.15.59:37522) 2024-03-19T03:07:11.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.810+0000 7effb2ffd700 1 -- 172.21.15.59:0/42121152 learned_addr learned my addr 172.21.15.59:0/42121152 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:11.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.810+0000 7effb2ffd700 1 -- 172.21.15.59:0/42121152 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effac0094f0 con 0x7effb4102bb0 2024-03-19T03:07:11.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.810+0000 7effb2ffd700 1 --2- 172.21.15.59:0/42121152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effb4102bb0 0x7effb4107880 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7effac006d40 tx=0x7effac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:11.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.810+0000 7effab7fe700 1 -- 172.21.15.59:0/42121152 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effac010840 con 0x7effb4102bb0 2024-03-19T03:07:11.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.810+0000 7effab7fe700 1 -- 172.21.15.59:0/42121152 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effac010e70 con 0x7effb4102bb0 2024-03-19T03:07:11.604 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.810+0000 7effab7fe700 1 -- 172.21.15.59:0/42121152 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effac024ea0 con 0x7effb4102bb0 2024-03-19T03:07:11.604 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.811+0000 7effb9917700 1 -- 172.21.15.59:0/42121152 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effb4105e80 con 0x7effb4102bb0 2024-03-19T03:07:11.604 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.811+0000 7effb9917700 1 -- 172.21.15.59:0/42121152 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effb4106370 con 0x7effb4102bb0 2024-03-19T03:07:11.604 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.812+0000 7effab7fe700 1 -- 172.21.15.59:0/42121152 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7effac01a4c0 con 0x7effb4102bb0 2024-03-19T03:07:11.604 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.812+0000 7effb9917700 1 -- 172.21.15.59:0/42121152 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effb404bed0 con 0x7effb4102bb0 2024-03-19T03:07:11.604 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.812+0000 7effab7fe700 1 --2- 172.21.15.59:0/42121152 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7effa0036790 0x7effa0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:11.604 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.812+0000 7effab7fe700 1 -- 172.21.15.59:0/42121152 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7effac04a820 con 0x7effb4102bb0 2024-03-19T03:07:11.604 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.812+0000 7effb27fc700 1 --2- 172.21.15.59:0/42121152 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7effa0036790 0x7effa0038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:11.604 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.812+0000 7effb27fc700 1 --2- 172.21.15.59:0/42121152 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7effa0036790 0x7effa0038c40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7eff9c009990 tx=0x7eff9c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:11.604 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:10.817+0000 7effab7fe700 1 -- 172.21.15.59:0/42121152 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7effac018940 con 0x7effb4102bb0 2024-03-19T03:07:11.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:11.006+0000 7effb9917700 1 -- 172.21.15.59:0/42121152 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7effb4102460 con 0x7effb4102bb0 2024-03-19T03:07:11.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:11.009+0000 7effab7fe700 1 -- 172.21.15.59:0/42121152 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7effac02d3a0 con 0x7effb4102bb0 2024-03-19T03:07:11.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:11.011+0000 7effb9917700 1 -- 172.21.15.59:0/42121152 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7effa0036790 msgr2=0x7effa0038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:11.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:11.011+0000 7effb9917700 1 --2- 172.21.15.59:0/42121152 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7effa0036790 0x7effa0038c40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7eff9c009990 tx=0x7eff9c006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:11.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:11.011+0000 7effb9917700 1 -- 172.21.15.59:0/42121152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effb4102bb0 msgr2=0x7effb4107880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:11.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:11.011+0000 7effb9917700 1 --2- 172.21.15.59:0/42121152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effb4102bb0 0x7effb4107880 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7effac006d40 tx=0x7effac009250 comp rx=0 tx=0).stop 2024-03-19T03:07:11.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:11.011+0000 7effb9917700 1 -- 172.21.15.59:0/42121152 shutdown_connections 2024-03-19T03:07:11.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:11.011+0000 7effb9917700 1 --2- 172.21.15.59:0/42121152 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7effa0036790 0x7effa0038c40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:11.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:11.012+0000 7effb9917700 1 --2- 172.21.15.59:0/42121152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effb4102bb0 0x7effb4107880 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:11.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:11.012+0000 7effb9917700 1 -- 172.21.15.59:0/42121152 >> 172.21.15.59:0/42121152 conn(0x7effb40fc500 msgr2=0x7effb40fd410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:11.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:11.012+0000 7effb9917700 1 -- 172.21.15.59:0/42121152 shutdown_connections 2024-03-19T03:07:11.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:11.012+0000 7effb9917700 1 -- 172.21.15.59:0/42121152 wait complete. 2024-03-19T03:07:12.213 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:12 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/42121152' entity='client.admin' 2024-03-19T03:07:12.790 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.011+0000 7fe06fc68700 1 Processor -- start 2024-03-19T03:07:12.791 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.011+0000 7fe06fc68700 1 -- start start 2024-03-19T03:07:12.791 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.011+0000 7fe06fc68700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe068076bc0 0x7fe068105310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:12.791 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.011+0000 7fe06fc68700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe068105850 con 0x7fe068076bc0 2024-03-19T03:07:12.791 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.011+0000 7fe06da04700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe068076bc0 0x7fe068105310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:12.791 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.011+0000 7fe06da04700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe068076bc0 0x7fe068105310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37538/0 (socket says 172.21.15.59:37538) 2024-03-19T03:07:12.791 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.011+0000 7fe06da04700 1 -- 172.21.15.59:0/1550922155 learned_addr learned my addr 172.21.15.59:0/1550922155 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:12.791 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.012+0000 7fe06da04700 1 -- 172.21.15.59:0/1550922155 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe068105990 con 0x7fe068076bc0 2024-03-19T03:07:12.791 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.012+0000 7fe06da04700 1 --2- 172.21.15.59:0/1550922155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe068076bc0 0x7fe068105310 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe0640098e0 tx=0x7fe064009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e68f519c7c346b5f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:12.791 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.012+0000 7fe06ca02700 1 -- 172.21.15.59:0/1550922155 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe06400b860 con 0x7fe068076bc0 2024-03-19T03:07:12.791 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.012+0000 7fe06ca02700 1 -- 172.21.15.59:0/1550922155 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe06400be90 con 0x7fe068076bc0 2024-03-19T03:07:12.791 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.012+0000 7fe06ca02700 1 -- 172.21.15.59:0/1550922155 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0640103c0 con 0x7fe068076bc0 2024-03-19T03:07:12.791 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.013+0000 7fe06fc68700 1 -- 172.21.15.59:0/1550922155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe068076bc0 msgr2=0x7fe068105310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:12.791 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.013+0000 7fe06fc68700 1 --2- 172.21.15.59:0/1550922155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe068076bc0 0x7fe068105310 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe0640098e0 tx=0x7fe064009250 comp rx=0 tx=0).stop 2024-03-19T03:07:12.792 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.013+0000 7fe06fc68700 1 -- 172.21.15.59:0/1550922155 shutdown_connections 2024-03-19T03:07:12.792 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.013+0000 7fe06fc68700 1 --2- 172.21.15.59:0/1550922155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe068076bc0 0x7fe068105310 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:12.792 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.013+0000 7fe06fc68700 1 -- 172.21.15.59:0/1550922155 >> 172.21.15.59:0/1550922155 conn(0x7fe0680fcd80 msgr2=0x7fe0680ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:12.792 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.013+0000 7fe06fc68700 1 -- 172.21.15.59:0/1550922155 shutdown_connections 2024-03-19T03:07:12.792 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.013+0000 7fe06fc68700 1 -- 172.21.15.59:0/1550922155 wait complete. 2024-03-19T03:07:12.792 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.014+0000 7fe06fc68700 1 Processor -- start 2024-03-19T03:07:12.792 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.014+0000 7fe06fc68700 1 -- start start 2024-03-19T03:07:12.792 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.014+0000 7fe06fc68700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe068076bc0 0x7fe068102c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:12.792 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.014+0000 7fe06fc68700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0681031a0 con 0x7fe068076bc0 2024-03-19T03:07:12.792 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.014+0000 7fe06da04700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe068076bc0 0x7fe068102c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:12.792 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.014+0000 7fe06da04700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe068076bc0 0x7fe068102c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37552/0 (socket says 172.21.15.59:37552) 2024-03-19T03:07:12.792 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.014+0000 7fe06da04700 1 -- 172.21.15.59:0/154913129 learned_addr learned my addr 172.21.15.59:0/154913129 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:12.792 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.014+0000 7fe06da04700 1 -- 172.21.15.59:0/154913129 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0640094f0 con 0x7fe068076bc0 2024-03-19T03:07:12.792 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.014+0000 7fe06da04700 1 --2- 172.21.15.59:0/154913129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe068076bc0 0x7fe068102c60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe064006d40 tx=0x7fe064009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:12.792 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.015+0000 7fe05e7fc700 1 -- 172.21.15.59:0/154913129 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe064010840 con 0x7fe068076bc0 2024-03-19T03:07:12.793 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.015+0000 7fe05e7fc700 1 -- 172.21.15.59:0/154913129 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe064010e70 con 0x7fe068076bc0 2024-03-19T03:07:12.793 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.015+0000 7fe06fc68700 1 -- 172.21.15.59:0/154913129 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe068101270 con 0x7fe068076bc0 2024-03-19T03:07:12.793 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.015+0000 7fe05e7fc700 1 -- 172.21.15.59:0/154913129 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe064024ea0 con 0x7fe068076bc0 2024-03-19T03:07:12.793 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.015+0000 7fe06fc68700 1 -- 172.21.15.59:0/154913129 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe068101760 con 0x7fe068076bc0 2024-03-19T03:07:12.793 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.015+0000 7fe05e7fc700 1 -- 172.21.15.59:0/154913129 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fe06401a4c0 con 0x7fe068076bc0 2024-03-19T03:07:12.793 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.016+0000 7fe05e7fc700 1 --2- 172.21.15.59:0/154913129 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fe054036790 0x7fe054038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:12.793 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.016+0000 7fe06fc68700 1 -- 172.21.15.59:0/154913129 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe06804bed0 con 0x7fe068076bc0 2024-03-19T03:07:12.793 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.016+0000 7fe05e7fc700 1 -- 172.21.15.59:0/154913129 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe06404a650 con 0x7fe068076bc0 2024-03-19T03:07:12.793 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.016+0000 7fe06d203700 1 --2- 172.21.15.59:0/154913129 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fe054036790 0x7fe054038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:12.793 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.016+0000 7fe06d203700 1 --2- 172.21.15.59:0/154913129 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fe054036790 0x7fe054038c40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe058009990 tx=0x7fe058006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:12.793 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.021+0000 7fe05e7fc700 1 -- 172.21.15.59:0/154913129 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fe064018940 con 0x7fe068076bc0 2024-03-19T03:07:12.793 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.211+0000 7fe06fc68700 1 -- 172.21.15.59:0/154913129 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fe068101c40 con 0x7fe068076bc0 2024-03-19T03:07:12.793 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.215+0000 7fe05e7fc700 1 -- 172.21.15.59:0/154913129 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fe064018350 con 0x7fe068076bc0 2024-03-19T03:07:12.793 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.217+0000 7fe06fc68700 1 -- 172.21.15.59:0/154913129 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fe054036790 msgr2=0x7fe054038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:12.794 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.217+0000 7fe06fc68700 1 --2- 172.21.15.59:0/154913129 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fe054036790 0x7fe054038c40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe058009990 tx=0x7fe058006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:12.794 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.217+0000 7fe06fc68700 1 -- 172.21.15.59:0/154913129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe068076bc0 msgr2=0x7fe068102c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:12.794 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.217+0000 7fe06fc68700 1 --2- 172.21.15.59:0/154913129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe068076bc0 0x7fe068102c60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe064006d40 tx=0x7fe064009250 comp rx=0 tx=0).stop 2024-03-19T03:07:12.794 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.217+0000 7fe06fc68700 1 -- 172.21.15.59:0/154913129 shutdown_connections 2024-03-19T03:07:12.794 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.217+0000 7fe06fc68700 1 --2- 172.21.15.59:0/154913129 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7fe054036790 0x7fe054038c40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:12.794 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.217+0000 7fe06fc68700 1 --2- 172.21.15.59:0/154913129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe068076bc0 0x7fe068102c60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:12.794 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.217+0000 7fe06fc68700 1 -- 172.21.15.59:0/154913129 >> 172.21.15.59:0/154913129 conn(0x7fe0680fcd80 msgr2=0x7fe0680fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:12.794 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.218+0000 7fe06fc68700 1 -- 172.21.15.59:0/154913129 shutdown_connections 2024-03-19T03:07:12.794 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:12.218+0000 7fe06fc68700 1 -- 172.21.15.59:0/154913129 wait complete. 2024-03-19T03:07:12.794 INFO:teuthology.orchestra.run.smithi059.stdout:Enabling the dashboard module... 2024-03-19T03:07:13.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:13 smithi059 ceph-mon[29133]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:07:13.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:13 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/154913129' entity='client.admin' 2024-03-19T03:07:14.330 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:14 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/2849225938' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-19T03:07:14.595 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.218+0000 7ff936e1a700 1 Processor -- start 2024-03-19T03:07:14.596 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.219+0000 7ff936e1a700 1 -- start start 2024-03-19T03:07:14.596 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.219+0000 7ff936e1a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff930078c50 0x7ff930077200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:14.596 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.219+0000 7ff936e1a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff930079060 con 0x7ff930078c50 2024-03-19T03:07:14.596 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.219+0000 7ff934bb6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff930078c50 0x7ff930077200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:14.596 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.219+0000 7ff934bb6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff930078c50 0x7ff930077200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37566/0 (socket says 172.21.15.59:37566) 2024-03-19T03:07:14.596 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.219+0000 7ff934bb6700 1 -- 172.21.15.59:0/954307608 learned_addr learned my addr 172.21.15.59:0/954307608 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:14.596 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.219+0000 7ff934bb6700 1 -- 172.21.15.59:0/954307608 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff930077740 con 0x7ff930078c50 2024-03-19T03:07:14.596 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.219+0000 7ff934bb6700 1 --2- 172.21.15.59:0/954307608 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff930078c50 0x7ff930077200 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff9240098e0 tx=0x7ff924009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e230770e2f0c34b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:14.596 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.219+0000 7ff92f7fe700 1 -- 172.21.15.59:0/954307608 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff92400b860 con 0x7ff930078c50 2024-03-19T03:07:14.596 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.220+0000 7ff92f7fe700 1 -- 172.21.15.59:0/954307608 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff92400be90 con 0x7ff930078c50 2024-03-19T03:07:14.596 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.220+0000 7ff92f7fe700 1 -- 172.21.15.59:0/954307608 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9240103c0 con 0x7ff930078c50 2024-03-19T03:07:14.596 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.220+0000 7ff936e1a700 1 -- 172.21.15.59:0/954307608 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff930078c50 msgr2=0x7ff930077200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:14.597 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.220+0000 7ff936e1a700 1 --2- 172.21.15.59:0/954307608 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff930078c50 0x7ff930077200 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff9240098e0 tx=0x7ff924009250 comp rx=0 tx=0).stop 2024-03-19T03:07:14.597 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.220+0000 7ff936e1a700 1 -- 172.21.15.59:0/954307608 shutdown_connections 2024-03-19T03:07:14.597 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.220+0000 7ff936e1a700 1 --2- 172.21.15.59:0/954307608 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff930078c50 0x7ff930077200 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:14.597 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.220+0000 7ff936e1a700 1 -- 172.21.15.59:0/954307608 >> 172.21.15.59:0/954307608 conn(0x7ff9300fde90 msgr2=0x7ff930100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:14.597 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.220+0000 7ff936e1a700 1 -- 172.21.15.59:0/954307608 shutdown_connections 2024-03-19T03:07:14.597 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.220+0000 7ff936e1a700 1 -- 172.21.15.59:0/954307608 wait complete. 2024-03-19T03:07:14.597 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.221+0000 7ff936e1a700 1 Processor -- start 2024-03-19T03:07:14.597 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.221+0000 7ff936e1a700 1 -- start start 2024-03-19T03:07:14.597 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.221+0000 7ff936e1a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff930078c50 0x7ff930113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:14.597 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.221+0000 7ff936e1a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff930113d60 con 0x7ff930078c50 2024-03-19T03:07:14.597 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.221+0000 7ff934bb6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff930078c50 0x7ff930113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:14.597 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.221+0000 7ff934bb6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff930078c50 0x7ff930113820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37580/0 (socket says 172.21.15.59:37580) 2024-03-19T03:07:14.597 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.221+0000 7ff934bb6700 1 -- 172.21.15.59:0/2849225938 learned_addr learned my addr 172.21.15.59:0/2849225938 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:14.597 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.221+0000 7ff934bb6700 1 -- 172.21.15.59:0/2849225938 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9240094f0 con 0x7ff930078c50 2024-03-19T03:07:14.597 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.222+0000 7ff934bb6700 1 --2- 172.21.15.59:0/2849225938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff930078c50 0x7ff930113820 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff9240094c0 tx=0x7ff924009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:14.598 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.222+0000 7ff92d7fa700 1 -- 172.21.15.59:0/2849225938 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff924010890 con 0x7ff930078c50 2024-03-19T03:07:14.598 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.222+0000 7ff92d7fa700 1 -- 172.21.15.59:0/2849225938 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff924010ec0 con 0x7ff930078c50 2024-03-19T03:07:14.598 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.222+0000 7ff936e1a700 1 -- 172.21.15.59:0/2849225938 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff930113f60 con 0x7ff930078c50 2024-03-19T03:07:14.598 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.222+0000 7ff92d7fa700 1 -- 172.21.15.59:0/2849225938 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9240243f0 con 0x7ff930078c50 2024-03-19T03:07:14.598 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.222+0000 7ff936e1a700 1 -- 172.21.15.59:0/2849225938 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff930114ad0 con 0x7ff930078c50 2024-03-19T03:07:14.598 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.223+0000 7ff92d7fa700 1 -- 172.21.15.59:0/2849225938 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7ff92402d3f0 con 0x7ff930078c50 2024-03-19T03:07:14.598 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.223+0000 7ff936e1a700 1 -- 172.21.15.59:0/2849225938 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff93005f5f0 con 0x7ff930078c50 2024-03-19T03:07:14.598 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.223+0000 7ff92d7fa700 1 --2- 172.21.15.59:0/2849225938 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7ff920036790 0x7ff920038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:14.598 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.223+0000 7ff92d7fa700 1 -- 172.21.15.59:0/2849225938 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff92404a570 con 0x7ff930078c50 2024-03-19T03:07:14.598 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.223+0000 7ff92ffff700 1 --2- 172.21.15.59:0/2849225938 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7ff920036790 0x7ff920038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:14.598 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.223+0000 7ff92ffff700 1 --2- 172.21.15.59:0/2849225938 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7ff920036790 0x7ff920038c40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff91c009990 tx=0x7ff91c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:14.598 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.228+0000 7ff92d7fa700 1 -- 172.21.15.59:0/2849225938 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7ff92402b8e0 con 0x7ff930078c50 2024-03-19T03:07:14.599 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:13.474+0000 7ff936e1a700 1 -- 172.21.15.59:0/2849225938 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7ff93004bed0 con 0x7ff930078c50 2024-03-19T03:07:14.599 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:14.017+0000 7ff92d7fa700 1 -- 172.21.15.59:0/2849225938 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7ff92401a7e0 con 0x7ff930078c50 2024-03-19T03:07:14.599 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:14.018+0000 7ff92d7fa700 1 -- 172.21.15.59:0/2849225938 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7ff92402d7b0 con 0x7ff930078c50 2024-03-19T03:07:14.599 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:14.022+0000 7ff936e1a700 1 -- 172.21.15.59:0/2849225938 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7ff920036790 msgr2=0x7ff920038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:14.599 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:14.022+0000 7ff936e1a700 1 --2- 172.21.15.59:0/2849225938 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7ff920036790 0x7ff920038c40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff91c009990 tx=0x7ff91c006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:14.599 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:14.022+0000 7ff936e1a700 1 -- 172.21.15.59:0/2849225938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff930078c50 msgr2=0x7ff930113820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:14.599 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:14.022+0000 7ff936e1a700 1 --2- 172.21.15.59:0/2849225938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff930078c50 0x7ff930113820 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff9240094c0 tx=0x7ff924009250 comp rx=0 tx=0).stop 2024-03-19T03:07:14.599 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:14.022+0000 7ff936e1a700 1 -- 172.21.15.59:0/2849225938 shutdown_connections 2024-03-19T03:07:14.599 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:14.022+0000 7ff936e1a700 1 --2- 172.21.15.59:0/2849225938 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7ff920036790 0x7ff920038c40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:14.599 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:14.022+0000 7ff936e1a700 1 --2- 172.21.15.59:0/2849225938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff930078c50 0x7ff930113820 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:14.599 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:14.022+0000 7ff936e1a700 1 -- 172.21.15.59:0/2849225938 >> 172.21.15.59:0/2849225938 conn(0x7ff9300fde90 msgr2=0x7ff9300feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:14.599 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:14.022+0000 7ff936e1a700 1 -- 172.21.15.59:0/2849225938 shutdown_connections 2024-03-19T03:07:14.599 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:14.022+0000 7ff936e1a700 1 -- 172.21.15.59:0/2849225938 wait complete. 2024-03-19T03:07:15.227 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:15 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/2849225938' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-19T03:07:15.228 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:15 smithi059 ceph-mon[29133]: mgrmap e9: smithi059.azpecc(active, since 24s) 2024-03-19T03:07:15.870 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2024-03-19T03:07:15.870 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-19T03:07:15.870 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-19T03:07:15.870 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "active_name": "smithi059.azpecc", 2024-03-19T03:07:15.871 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-19T03:07:15.871 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2024-03-19T03:07:15.871 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.020+0000 7f0325f32700 1 Processor -- start 2024-03-19T03:07:15.871 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.020+0000 7f0325f32700 1 -- start start 2024-03-19T03:07:15.871 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.020+0000 7f0325f32700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0320071360 0x7f03201052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:15.871 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.020+0000 7f0325f32700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0320105820 con 0x7f0320071360 2024-03-19T03:07:15.871 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.020+0000 7f031f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0320071360 0x7f03201052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:15.871 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.020+0000 7f031f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0320071360 0x7f03201052e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58508/0 (socket says 172.21.15.59:58508) 2024-03-19T03:07:15.871 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.020+0000 7f031f7fe700 1 -- 172.21.15.59:0/836034261 learned_addr learned my addr 172.21.15.59:0/836034261 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:15.871 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.021+0000 7f031f7fe700 1 -- 172.21.15.59:0/836034261 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0320105960 con 0x7f0320071360 2024-03-19T03:07:15.871 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.021+0000 7f031f7fe700 1 --2- 172.21.15.59:0/836034261 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0320071360 0x7f03201052e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f03080098e0 tx=0x7f0308009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d08e7fe534a68d9c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:15.871 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.021+0000 7f031e7fc700 1 -- 172.21.15.59:0/836034261 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f030800b860 con 0x7f0320071360 2024-03-19T03:07:15.871 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.021+0000 7f031e7fc700 1 -- 172.21.15.59:0/836034261 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f030800be90 con 0x7f0320071360 2024-03-19T03:07:15.871 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.021+0000 7f031e7fc700 1 -- 172.21.15.59:0/836034261 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03080103c0 con 0x7f0320071360 2024-03-19T03:07:15.872 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.022+0000 7f0325f32700 1 -- 172.21.15.59:0/836034261 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0320071360 msgr2=0x7f03201052e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:15.872 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.022+0000 7f0325f32700 1 --2- 172.21.15.59:0/836034261 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0320071360 0x7f03201052e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f03080098e0 tx=0x7f0308009250 comp rx=0 tx=0).stop 2024-03-19T03:07:15.872 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.022+0000 7f0325f32700 1 -- 172.21.15.59:0/836034261 shutdown_connections 2024-03-19T03:07:15.872 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.022+0000 7f0325f32700 1 --2- 172.21.15.59:0/836034261 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0320071360 0x7f03201052e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:15.872 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.022+0000 7f0325f32700 1 -- 172.21.15.59:0/836034261 >> 172.21.15.59:0/836034261 conn(0x7f03200fcd50 msgr2=0x7f03200ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:15.872 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.022+0000 7f0325f32700 1 -- 172.21.15.59:0/836034261 shutdown_connections 2024-03-19T03:07:15.872 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.022+0000 7f0325f32700 1 -- 172.21.15.59:0/836034261 wait complete. 2024-03-19T03:07:15.872 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.022+0000 7f0325f32700 1 Processor -- start 2024-03-19T03:07:15.872 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.022+0000 7f0325f32700 1 -- start start 2024-03-19T03:07:15.872 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.023+0000 7f0325f32700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0320071360 0x7f0320111640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:15.872 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.023+0000 7f0325f32700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0320111b80 con 0x7f0320071360 2024-03-19T03:07:15.872 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.023+0000 7f031f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0320071360 0x7f0320111640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:15.873 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.023+0000 7f031f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0320071360 0x7f0320111640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58520/0 (socket says 172.21.15.59:58520) 2024-03-19T03:07:15.873 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.023+0000 7f031f7fe700 1 -- 172.21.15.59:0/1225175914 learned_addr learned my addr 172.21.15.59:0/1225175914 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:15.873 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.023+0000 7f031f7fe700 1 -- 172.21.15.59:0/1225175914 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03080094f0 con 0x7f0320071360 2024-03-19T03:07:15.873 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.023+0000 7f031f7fe700 1 --2- 172.21.15.59:0/1225175914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0320071360 0x7f0320111640 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f03080105b0 tx=0x7f0308009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:15.873 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.023+0000 7f0317fff700 1 -- 172.21.15.59:0/1225175914 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0308010840 con 0x7f0320071360 2024-03-19T03:07:15.873 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.023+0000 7f0317fff700 1 -- 172.21.15.59:0/1225175914 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0308010e70 con 0x7f0320071360 2024-03-19T03:07:15.873 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.023+0000 7f0325f32700 1 -- 172.21.15.59:0/1225175914 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0320111d80 con 0x7f0320071360 2024-03-19T03:07:15.873 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.023+0000 7f0317fff700 1 -- 172.21.15.59:0/1225175914 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f030801ae30 con 0x7f0320071360 2024-03-19T03:07:15.873 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.023+0000 7f0325f32700 1 -- 172.21.15.59:0/1225175914 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03201128f0 con 0x7f0320071360 2024-03-19T03:07:15.873 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.024+0000 7f0317fff700 1 -- 172.21.15.59:0/1225175914 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f030802d3f0 con 0x7f0320071360 2024-03-19T03:07:15.873 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.024+0000 7f0325f32700 1 -- 172.21.15.59:0/1225175914 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f032005f5f0 con 0x7f0320071360 2024-03-19T03:07:15.873 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.024+0000 7f0317fff700 1 --2- 172.21.15.59:0/1225175914 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f030c0367e0 0x7f030c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:15.873 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.025+0000 7f031effd700 1 -- 172.21.15.59:0/1225175914 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f030c0367e0 msgr2=0x7f030c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1573642037 2024-03-19T03:07:15.873 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.025+0000 7f0317fff700 1 -- 172.21.15.59:0/1225175914 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f030804a250 con 0x7f0320071360 2024-03-19T03:07:15.874 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.025+0000 7f031effd700 1 --2- 172.21.15.59:0/1225175914 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f030c0367e0 0x7f030c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T03:07:15.874 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.029+0000 7f0317fff700 1 -- 172.21.15.59:0/1225175914 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f030802b4d0 con 0x7f0320071360 2024-03-19T03:07:15.874 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.226+0000 7f031effd700 1 -- 172.21.15.59:0/1225175914 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f030c0367e0 msgr2=0x7f030c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1573642037 2024-03-19T03:07:15.874 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.226+0000 7f031effd700 1 --2- 172.21.15.59:0/1225175914 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f030c0367e0 0x7f030c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T03:07:15.874 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.278+0000 7f0325f32700 1 -- 172.21.15.59:0/1225175914 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f0320112c90 con 0x7f0320071360 2024-03-19T03:07:15.874 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.279+0000 7f0317fff700 1 -- 172.21.15.59:0/1225175914 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f030802c9f0 con 0x7f0320071360 2024-03-19T03:07:15.874 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.281+0000 7f0325f32700 1 -- 172.21.15.59:0/1225175914 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f030c0367e0 msgr2=0x7f030c038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T03:07:15.874 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.281+0000 7f0325f32700 1 --2- 172.21.15.59:0/1225175914 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f030c0367e0 0x7f030c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:15.874 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.281+0000 7f0325f32700 1 -- 172.21.15.59:0/1225175914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0320071360 msgr2=0x7f0320111640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:15.874 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.281+0000 7f0325f32700 1 --2- 172.21.15.59:0/1225175914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0320071360 0x7f0320111640 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f03080105b0 tx=0x7f0308009250 comp rx=0 tx=0).stop 2024-03-19T03:07:15.874 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.281+0000 7f0325f32700 1 -- 172.21.15.59:0/1225175914 shutdown_connections 2024-03-19T03:07:15.874 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.281+0000 7f0325f32700 1 --2- 172.21.15.59:0/1225175914 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f030c0367e0 0x7f030c038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:15.874 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.281+0000 7f0325f32700 1 --2- 172.21.15.59:0/1225175914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0320071360 0x7f0320111640 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:15.874 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.281+0000 7f0325f32700 1 -- 172.21.15.59:0/1225175914 >> 172.21.15.59:0/1225175914 conn(0x7f03200fcd50 msgr2=0x7f03200fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:15.875 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.281+0000 7f0325f32700 1 -- 172.21.15.59:0/1225175914 shutdown_connections 2024-03-19T03:07:15.875 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:15.281+0000 7f0325f32700 1 -- 172.21.15.59:0/1225175914 wait complete. 2024-03-19T03:07:15.875 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for the mgr to restart... 2024-03-19T03:07:15.875 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for mgr epoch 9... 2024-03-19T03:07:16.113 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:16 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/1225175914' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-19T03:07:21.452 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:21 smithi059 ceph-mon[29133]: Active manager daemon smithi059.azpecc restarted 2024-03-19T03:07:21.452 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:21 smithi059 ceph-mon[29133]: Activating manager daemon smithi059.azpecc 2024-03-19T03:07:21.452 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:21 smithi059 ceph-mon[29133]: osdmap e3: 0 total, 0 up, 0 in 2024-03-19T03:07:21.452 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:21 smithi059 ceph-mon[29133]: mgrmap e10: smithi059.azpecc(active, starting, since 0.00371195s) 2024-03-19T03:07:21.452 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:21 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-03-19T03:07:21.452 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:21 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mgr metadata", "who": "smithi059.azpecc", "id": "smithi059.azpecc"}]: dispatch 2024-03-19T03:07:21.452 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:21 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T03:07:21.452 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:21 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T03:07:21.452 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:21 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T03:07:21.452 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:21 smithi059 ceph-mon[29133]: Manager daemon smithi059.azpecc is now available 2024-03-19T03:07:21.453 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:21 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:07:21.453 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:21 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:07:21.453 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:21 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.azpecc/mirror_snapshot_schedule"}]: dispatch 2024-03-19T03:07:21.453 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:21 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.azpecc/trash_purge_schedule"}]: dispatch 2024-03-19T03:07:22.484 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2024-03-19T03:07:22.485 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-19T03:07:22.485 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-19T03:07:22.485 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2024-03-19T03:07:22.485 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.310+0000 7f4789eaa700 1 Processor -- start 2024-03-19T03:07:22.485 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.310+0000 7f4789eaa700 1 -- start start 2024-03-19T03:07:22.485 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.310+0000 7f4789eaa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4784102be0 0x7f4784102ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:22.485 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.310+0000 7f4789eaa700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4784103530 con 0x7f4784102be0 2024-03-19T03:07:22.485 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.310+0000 7f47837fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4784102be0 0x7f4784102ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:22.485 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.310+0000 7f47837fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4784102be0 0x7f4784102ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58526/0 (socket says 172.21.15.59:58526) 2024-03-19T03:07:22.485 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.310+0000 7f47837fe700 1 -- 172.21.15.59:0/3193879461 learned_addr learned my addr 172.21.15.59:0/3193879461 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:22.485 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.311+0000 7f47837fe700 1 -- 172.21.15.59:0/3193879461 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4784103670 con 0x7f4784102be0 2024-03-19T03:07:22.485 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.311+0000 7f47837fe700 1 --2- 172.21.15.59:0/3193879461 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4784102be0 0x7f4784102ff0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f47740098e0 tx=0x7f4774009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc62379bf6f7886 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:22.485 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.311+0000 7f47827fc700 1 -- 172.21.15.59:0/3193879461 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f477400b860 con 0x7f4784102be0 2024-03-19T03:07:22.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.311+0000 7f47827fc700 1 -- 172.21.15.59:0/3193879461 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f477400be90 con 0x7f4784102be0 2024-03-19T03:07:22.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.311+0000 7f47827fc700 1 -- 172.21.15.59:0/3193879461 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47740103c0 con 0x7f4784102be0 2024-03-19T03:07:22.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.312+0000 7f4789eaa700 1 -- 172.21.15.59:0/3193879461 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4784102be0 msgr2=0x7f4784102ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:22.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.312+0000 7f4789eaa700 1 --2- 172.21.15.59:0/3193879461 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4784102be0 0x7f4784102ff0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f47740098e0 tx=0x7f4774009250 comp rx=0 tx=0).stop 2024-03-19T03:07:22.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.312+0000 7f4789eaa700 1 -- 172.21.15.59:0/3193879461 shutdown_connections 2024-03-19T03:07:22.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.312+0000 7f4789eaa700 1 --2- 172.21.15.59:0/3193879461 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4784102be0 0x7f4784102ff0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:22.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.312+0000 7f4789eaa700 1 -- 172.21.15.59:0/3193879461 >> 172.21.15.59:0/3193879461 conn(0x7f47840fc500 msgr2=0x7f47840fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:22.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.312+0000 7f4789eaa700 1 -- 172.21.15.59:0/3193879461 shutdown_connections 2024-03-19T03:07:22.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.312+0000 7f4789eaa700 1 -- 172.21.15.59:0/3193879461 wait complete. 2024-03-19T03:07:22.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.312+0000 7f4789eaa700 1 Processor -- start 2024-03-19T03:07:22.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.312+0000 7f4789eaa700 1 -- start start 2024-03-19T03:07:22.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.312+0000 7f4789eaa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4784102be0 0x7f47841013f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:22.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.312+0000 7f4789eaa700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4784114240 con 0x7f4784102be0 2024-03-19T03:07:22.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.313+0000 7f47837fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4784102be0 0x7f47841013f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:22.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.313+0000 7f47837fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4784102be0 0x7f47841013f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58538/0 (socket says 172.21.15.59:58538) 2024-03-19T03:07:22.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.313+0000 7f47837fe700 1 -- 172.21.15.59:0/3966239588 learned_addr learned my addr 172.21.15.59:0/3966239588 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:22.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.313+0000 7f47837fe700 1 -- 172.21.15.59:0/3966239588 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47740094f0 con 0x7f4784102be0 2024-03-19T03:07:22.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.313+0000 7f47837fe700 1 --2- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4784102be0 0x7f47841013f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4774011680 tx=0x7f4774009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:22.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.313+0000 7f476bfff700 1 -- 172.21.15.59:0/3966239588 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4774010890 con 0x7f4784102be0 2024-03-19T03:07:22.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.313+0000 7f476bfff700 1 -- 172.21.15.59:0/3966239588 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4774010ec0 con 0x7f4784102be0 2024-03-19T03:07:22.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.313+0000 7f476bfff700 1 -- 172.21.15.59:0/3966239588 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47740243f0 con 0x7f4784102be0 2024-03-19T03:07:22.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.313+0000 7f4789eaa700 1 -- 172.21.15.59:0/3966239588 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4784114380 con 0x7f4784102be0 2024-03-19T03:07:22.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.313+0000 7f4789eaa700 1 -- 172.21.15.59:0/3966239588 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4784114870 con 0x7f4784102be0 2024-03-19T03:07:22.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.314+0000 7f476bfff700 1 -- 172.21.15.59:0/3966239588 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f477402d3f0 con 0x7f4784102be0 2024-03-19T03:07:22.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.314+0000 7f476bfff700 1 --2- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f476c036830 0x7f476c038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:22.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.314+0000 7f4789eaa700 1 -- 172.21.15.59:0/3966239588 --> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f478404bed0 con 0x7f476c036830 2024-03-19T03:07:22.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.314+0000 7f476bfff700 1 -- 172.21.15.59:0/3966239588 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f477404a2a0 con 0x7f4784102be0 2024-03-19T03:07:22.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.314+0000 7f4782ffd700 1 -- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f476c036830 msgr2=0x7f476c038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1573642037 2024-03-19T03:07:22.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.314+0000 7f4782ffd700 1 --2- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f476c036830 0x7f476c038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T03:07:22.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.515+0000 7f4782ffd700 1 -- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f476c036830 msgr2=0x7f476c038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1573642037 2024-03-19T03:07:22.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.515+0000 7f4782ffd700 1 --2- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f476c036830 0x7f476c038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T03:07:22.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.916+0000 7f4782ffd700 1 -- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f476c036830 msgr2=0x7f476c038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1573642037 2024-03-19T03:07:22.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:16.916+0000 7f4782ffd700 1 --2- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f476c036830 0x7f476c038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T03:07:22.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:17.717+0000 7f4782ffd700 1 -- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f476c036830 msgr2=0x7f476c038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1573642037 2024-03-19T03:07:22.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:17.717+0000 7f4782ffd700 1 --2- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f476c036830 0x7f476c038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T03:07:22.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:19.318+0000 7f4782ffd700 1 -- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f476c036830 msgr2=0x7f476c038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1573642037 2024-03-19T03:07:22.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:19.318+0000 7f4782ffd700 1 --2- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f476c036830 0x7f476c038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-19T03:07:22.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:20.739+0000 7f476bfff700 1 -- 172.21.15.59:0/3966239588 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mgrmap(e 10) v1 ==== 43148+0+0 (secure 0 0 0) 0x7f477401a8c0 con 0x7f4784102be0 2024-03-19T03:07:22.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:20.739+0000 7f476bfff700 1 -- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f476c036830 msgr2=0x7f476c038ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T03:07:22.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:20.739+0000 7f476bfff700 1 --2- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f476c036830 0x7f476c038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:22.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:21.742+0000 7f476bfff700 1 -- 172.21.15.59:0/3966239588 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f4774029d00 con 0x7f4784102be0 2024-03-19T03:07:22.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:21.742+0000 7f476bfff700 1 --2- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f476c039ff0 0x7f476c03c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:22.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:21.742+0000 7f476bfff700 1 -- 172.21.15.59:0/3966239588 --> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f478404bed0 con 0x7f476c039ff0 2024-03-19T03:07:22.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:21.742+0000 7f4782ffd700 1 --2- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f476c039ff0 0x7f476c03c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:22.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:21.743+0000 7f4782ffd700 1 --2- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f476c039ff0 0x7f476c03c3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f47780039d0 tx=0x7f4778009350 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:22.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:21.743+0000 7f476bfff700 1 -- 172.21.15.59:0/3966239588 <== mgr.14162 v2:172.21.15.59:6800/1857243897 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f478404bed0 con 0x7f476c039ff0 2024-03-19T03:07:22.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:21.749+0000 7f4789eaa700 1 -- 172.21.15.59:0/3966239588 --> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f4784102420 con 0x7f476c039ff0 2024-03-19T03:07:22.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:21.750+0000 7f476bfff700 1 -- 172.21.15.59:0/3966239588 <== mgr.14162 v2:172.21.15.59:6800/1857243897 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f4784102420 con 0x7f476c039ff0 2024-03-19T03:07:22.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:21.750+0000 7f4789eaa700 1 -- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f476c039ff0 msgr2=0x7f476c03c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:22.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:21.750+0000 7f4789eaa700 1 --2- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f476c039ff0 0x7f476c03c3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f47780039d0 tx=0x7f4778009350 comp rx=0 tx=0).stop 2024-03-19T03:07:22.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:21.750+0000 7f4789eaa700 1 -- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4784102be0 msgr2=0x7f47841013f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:22.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:21.750+0000 7f4789eaa700 1 --2- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4784102be0 0x7f47841013f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4774011680 tx=0x7f4774009250 comp rx=0 tx=0).stop 2024-03-19T03:07:22.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:21.750+0000 7f4789eaa700 1 -- 172.21.15.59:0/3966239588 shutdown_connections 2024-03-19T03:07:22.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:21.750+0000 7f4789eaa700 1 --2- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f476c039ff0 0x7f476c03c3d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:22.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:21.750+0000 7f4789eaa700 1 --2- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4784102be0 0x7f47841013f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:22.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:21.750+0000 7f4789eaa700 1 --2- 172.21.15.59:0/3966239588 >> [v2:172.21.15.59:6800/1573642037,v1:172.21.15.59:6801/1573642037] conn(0x7f476c036830 0x7f476c038ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:22.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:21.750+0000 7f4789eaa700 1 -- 172.21.15.59:0/3966239588 >> 172.21.15.59:0/3966239588 conn(0x7f47840fc500 msgr2=0x7f47840fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:22.490 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:21.750+0000 7f4789eaa700 1 -- 172.21.15.59:0/3966239588 shutdown_connections 2024-03-19T03:07:22.490 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:21.750+0000 7f4789eaa700 1 -- 172.21.15.59:0/3966239588 wait complete. 2024-03-19T03:07:22.490 INFO:teuthology.orchestra.run.smithi059.stdout:mgr epoch 9 is available 2024-03-19T03:07:22.490 INFO:teuthology.orchestra.run.smithi059.stdout:Generating a dashboard self-signed certificate... 2024-03-19T03:07:22.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:22 smithi059 ceph-mon[29133]: [19/Mar/2024:03:07:21] ENGINE Bus STARTING 2024-03-19T03:07:22.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:22 smithi059 ceph-mon[29133]: [19/Mar/2024:03:07:21] ENGINE Serving on https://172.21.15.59:7150 2024-03-19T03:07:22.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:22 smithi059 ceph-mon[29133]: [19/Mar/2024:03:07:21] ENGINE Bus STARTED 2024-03-19T03:07:22.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:22 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:22.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:22 smithi059 ceph-mon[29133]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-19T03:07:22.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:22 smithi059 ceph-mon[29133]: mgrmap e11: smithi059.azpecc(active, since 1.00605s) 2024-03-19T03:07:22.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:22 smithi059 ceph-mon[29133]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-19T03:07:23.813 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-19T03:07:23.813 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.923+0000 7fd808164700 1 Processor -- start 2024-03-19T03:07:23.813 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.923+0000 7fd808164700 1 -- start start 2024-03-19T03:07:23.814 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.924+0000 7fd808164700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd800071360 0x7fd8001052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:23.814 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.924+0000 7fd808164700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd800105820 con 0x7fd800071360 2024-03-19T03:07:23.814 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.924+0000 7fd805f00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd800071360 0x7fd8001052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:23.814 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.924+0000 7fd805f00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd800071360 0x7fd8001052e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58606/0 (socket says 172.21.15.59:58606) 2024-03-19T03:07:23.814 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.924+0000 7fd805f00700 1 -- 172.21.15.59:0/1290333878 learned_addr learned my addr 172.21.15.59:0/1290333878 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:23.814 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.924+0000 7fd805f00700 1 -- 172.21.15.59:0/1290333878 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd800105960 con 0x7fd800071360 2024-03-19T03:07:23.814 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.924+0000 7fd805f00700 1 --2- 172.21.15.59:0/1290333878 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd800071360 0x7fd8001052e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd7f00098e0 tx=0x7fd7f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=819b76b884b30b52 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:23.814 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.924+0000 7fd804efe700 1 -- 172.21.15.59:0/1290333878 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7f000b860 con 0x7fd800071360 2024-03-19T03:07:23.814 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.925+0000 7fd804efe700 1 -- 172.21.15.59:0/1290333878 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd7f000be90 con 0x7fd800071360 2024-03-19T03:07:23.814 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.925+0000 7fd804efe700 1 -- 172.21.15.59:0/1290333878 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7f00103c0 con 0x7fd800071360 2024-03-19T03:07:23.814 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.925+0000 7fd808164700 1 -- 172.21.15.59:0/1290333878 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd800071360 msgr2=0x7fd8001052e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:23.814 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.925+0000 7fd808164700 1 --2- 172.21.15.59:0/1290333878 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd800071360 0x7fd8001052e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd7f00098e0 tx=0x7fd7f0009250 comp rx=0 tx=0).stop 2024-03-19T03:07:23.815 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.925+0000 7fd808164700 1 -- 172.21.15.59:0/1290333878 shutdown_connections 2024-03-19T03:07:23.815 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.925+0000 7fd808164700 1 --2- 172.21.15.59:0/1290333878 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd800071360 0x7fd8001052e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:23.815 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.925+0000 7fd808164700 1 -- 172.21.15.59:0/1290333878 >> 172.21.15.59:0/1290333878 conn(0x7fd8000fcd50 msgr2=0x7fd8000ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:23.815 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.925+0000 7fd808164700 1 -- 172.21.15.59:0/1290333878 shutdown_connections 2024-03-19T03:07:23.815 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.925+0000 7fd808164700 1 -- 172.21.15.59:0/1290333878 wait complete. 2024-03-19T03:07:23.815 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.926+0000 7fd808164700 1 Processor -- start 2024-03-19T03:07:23.815 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.926+0000 7fd808164700 1 -- start start 2024-03-19T03:07:23.815 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.926+0000 7fd808164700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd800071360 0x7fd800111640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:23.815 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.926+0000 7fd808164700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd800111b80 con 0x7fd800071360 2024-03-19T03:07:23.815 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.926+0000 7fd805f00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd800071360 0x7fd800111640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:23.815 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.926+0000 7fd805f00700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd800071360 0x7fd800111640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58622/0 (socket says 172.21.15.59:58622) 2024-03-19T03:07:23.815 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.926+0000 7fd805f00700 1 -- 172.21.15.59:0/2344461082 learned_addr learned my addr 172.21.15.59:0/2344461082 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:23.815 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.926+0000 7fd805f00700 1 -- 172.21.15.59:0/2344461082 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f00094f0 con 0x7fd800071360 2024-03-19T03:07:23.815 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.926+0000 7fd805f00700 1 --2- 172.21.15.59:0/2344461082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd800071360 0x7fd800111640 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd7f00094c0 tx=0x7fd7f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:23.816 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.927+0000 7fd7f67fc700 1 -- 172.21.15.59:0/2344461082 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7f00118f0 con 0x7fd800071360 2024-03-19T03:07:23.816 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.927+0000 7fd7f67fc700 1 -- 172.21.15.59:0/2344461082 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd7f0010d60 con 0x7fd800071360 2024-03-19T03:07:23.816 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.927+0000 7fd7f67fc700 1 -- 172.21.15.59:0/2344461082 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7f001a430 con 0x7fd800071360 2024-03-19T03:07:23.816 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.927+0000 7fd808164700 1 -- 172.21.15.59:0/2344461082 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd800111d80 con 0x7fd800071360 2024-03-19T03:07:23.816 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.927+0000 7fd808164700 1 -- 172.21.15.59:0/2344461082 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8001128f0 con 0x7fd800071360 2024-03-19T03:07:23.816 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.928+0000 7fd7f67fc700 1 -- 172.21.15.59:0/2344461082 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7fd7f0024440 con 0x7fd800071360 2024-03-19T03:07:23.816 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.928+0000 7fd808164700 1 -- 172.21.15.59:0/2344461082 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd80005f5f0 con 0x7fd800071360 2024-03-19T03:07:23.816 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.928+0000 7fd7f67fc700 1 --2- 172.21.15.59:0/2344461082 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fd7ec036710 0x7fd7ec038bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:23.816 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.928+0000 7fd7f67fc700 1 -- 172.21.15.59:0/2344461082 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd7f004a590 con 0x7fd800071360 2024-03-19T03:07:23.816 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.928+0000 7fd8056ff700 1 --2- 172.21.15.59:0/2344461082 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fd7ec036710 0x7fd7ec038bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:23.816 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.928+0000 7fd8056ff700 1 --2- 172.21.15.59:0/2344461082 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fd7ec036710 0x7fd7ec038bc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc009990 tx=0x7fd7fc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:23.817 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:22.933+0000 7fd7f67fc700 1 -- 172.21.15.59:0/2344461082 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fd7f0023de0 con 0x7fd800071360 2024-03-19T03:07:23.817 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:23.147+0000 7fd808164700 1 -- 172.21.15.59:0/2344461082 --> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8000fe140 con 0x7fd7ec036710 2024-03-19T03:07:23.817 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:23.219+0000 7fd7f67fc700 1 -- 172.21.15.59:0/2344461082 <== mgr.14162 v2:172.21.15.59:6800/1857243897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fd8000fe140 con 0x7fd7ec036710 2024-03-19T03:07:23.817 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:23.221+0000 7fd808164700 1 -- 172.21.15.59:0/2344461082 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fd7ec036710 msgr2=0x7fd7ec038bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:23.817 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:23.221+0000 7fd808164700 1 --2- 172.21.15.59:0/2344461082 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fd7ec036710 0x7fd7ec038bc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc009990 tx=0x7fd7fc006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:23.817 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:23.221+0000 7fd808164700 1 -- 172.21.15.59:0/2344461082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd800071360 msgr2=0x7fd800111640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:23.817 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:23.221+0000 7fd808164700 1 --2- 172.21.15.59:0/2344461082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd800071360 0x7fd800111640 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd7f00094c0 tx=0x7fd7f0009250 comp rx=0 tx=0).stop 2024-03-19T03:07:23.817 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:23.222+0000 7fd808164700 1 -- 172.21.15.59:0/2344461082 shutdown_connections 2024-03-19T03:07:23.817 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:23.222+0000 7fd808164700 1 --2- 172.21.15.59:0/2344461082 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fd7ec036710 0x7fd7ec038bc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:23.817 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:23.222+0000 7fd808164700 1 --2- 172.21.15.59:0/2344461082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd800071360 0x7fd800111640 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:23.818 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:23.222+0000 7fd808164700 1 -- 172.21.15.59:0/2344461082 >> 172.21.15.59:0/2344461082 conn(0x7fd8000fcd50 msgr2=0x7fd8000fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:23.818 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:23.222+0000 7fd808164700 1 -- 172.21.15.59:0/2344461082 shutdown_connections 2024-03-19T03:07:23.818 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:23.222+0000 7fd808164700 1 -- 172.21.15.59:0/2344461082 wait complete. 2024-03-19T03:07:23.818 INFO:teuthology.orchestra.run.smithi059.stdout:Creating initial admin user... 2024-03-19T03:07:24.191 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:24 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:24.192 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:24 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:24.192 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:24 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:24.192 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:24 smithi059 ceph-mon[29133]: mgrmap e12: smithi059.azpecc(active, since 2s) 2024-03-19T03:07:24.192 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:24 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:25.315 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:25 smithi059 ceph-mon[29133]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:07:25.315 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:25 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:25.560 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$1xGHkq0HEeLzmW83DqoTmO4Ym677b4WPsiaoBAkqrqEWqdFgx4szC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710817644, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-19T03:07:25.567 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.290+0000 7f5a76f68700 1 Processor -- start 2024-03-19T03:07:25.567 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.290+0000 7f5a76f68700 1 -- start start 2024-03-19T03:07:25.567 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.290+0000 7f5a76f68700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a70102720 0x7f5a70102b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:25.567 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.290+0000 7f5a76f68700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a70103070 con 0x7f5a70102720 2024-03-19T03:07:25.567 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.291+0000 7f5a74d04700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a70102720 0x7f5a70102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:25.568 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.291+0000 7f5a74d04700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a70102720 0x7f5a70102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33030/0 (socket says 172.21.15.59:33030) 2024-03-19T03:07:25.568 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.291+0000 7f5a74d04700 1 -- 172.21.15.59:0/3534379956 learned_addr learned my addr 172.21.15.59:0/3534379956 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:25.568 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.291+0000 7f5a74d04700 1 -- 172.21.15.59:0/3534379956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a701031b0 con 0x7f5a70102720 2024-03-19T03:07:25.568 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.291+0000 7f5a74d04700 1 --2- 172.21.15.59:0/3534379956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a70102720 0x7f5a70102b30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5a640098e0 tx=0x7f5a64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=723e95170c639b14 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:25.568 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.291+0000 7f5a6f7fe700 1 -- 172.21.15.59:0/3534379956 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a6400b860 con 0x7f5a70102720 2024-03-19T03:07:25.568 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.291+0000 7f5a6f7fe700 1 -- 172.21.15.59:0/3534379956 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a6400be90 con 0x7f5a70102720 2024-03-19T03:07:25.568 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.291+0000 7f5a6f7fe700 1 -- 172.21.15.59:0/3534379956 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a640103c0 con 0x7f5a70102720 2024-03-19T03:07:25.568 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.292+0000 7f5a76f68700 1 -- 172.21.15.59:0/3534379956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a70102720 msgr2=0x7f5a70102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:25.568 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.292+0000 7f5a76f68700 1 --2- 172.21.15.59:0/3534379956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a70102720 0x7f5a70102b30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5a640098e0 tx=0x7f5a64009250 comp rx=0 tx=0).stop 2024-03-19T03:07:25.568 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.292+0000 7f5a76f68700 1 -- 172.21.15.59:0/3534379956 shutdown_connections 2024-03-19T03:07:25.568 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.292+0000 7f5a76f68700 1 --2- 172.21.15.59:0/3534379956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a70102720 0x7f5a70102b30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:25.568 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.292+0000 7f5a76f68700 1 -- 172.21.15.59:0/3534379956 >> 172.21.15.59:0/3534379956 conn(0x7f5a700fde40 msgr2=0x7f5a701002b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:25.568 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.292+0000 7f5a76f68700 1 -- 172.21.15.59:0/3534379956 shutdown_connections 2024-03-19T03:07:25.569 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.292+0000 7f5a76f68700 1 -- 172.21.15.59:0/3534379956 wait complete. 2024-03-19T03:07:25.569 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.293+0000 7f5a76f68700 1 Processor -- start 2024-03-19T03:07:25.569 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.293+0000 7f5a76f68700 1 -- start start 2024-03-19T03:07:25.569 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.293+0000 7f5a76f68700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a70102720 0x7f5a701137a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:25.569 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.293+0000 7f5a76f68700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a70113ce0 con 0x7f5a70102720 2024-03-19T03:07:25.569 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.293+0000 7f5a74d04700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a70102720 0x7f5a701137a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:25.569 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.293+0000 7f5a74d04700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a70102720 0x7f5a701137a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33042/0 (socket says 172.21.15.59:33042) 2024-03-19T03:07:25.569 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.293+0000 7f5a74d04700 1 -- 172.21.15.59:0/2938197409 learned_addr learned my addr 172.21.15.59:0/2938197409 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:25.570 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.293+0000 7f5a74d04700 1 -- 172.21.15.59:0/2938197409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a640094f0 con 0x7f5a70102720 2024-03-19T03:07:25.570 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.293+0000 7f5a74d04700 1 --2- 172.21.15.59:0/2938197409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a70102720 0x7f5a701137a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5a640090b0 tx=0x7f5a64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:25.570 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.294+0000 7f5a6d7fa700 1 -- 172.21.15.59:0/2938197409 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a64010840 con 0x7f5a70102720 2024-03-19T03:07:25.570 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.294+0000 7f5a6d7fa700 1 -- 172.21.15.59:0/2938197409 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a64010e70 con 0x7f5a70102720 2024-03-19T03:07:25.570 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.294+0000 7f5a6d7fa700 1 -- 172.21.15.59:0/2938197409 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a64024e40 con 0x7f5a70102720 2024-03-19T03:07:25.570 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.294+0000 7f5a76f68700 1 -- 172.21.15.59:0/2938197409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a70113ee0 con 0x7f5a70102720 2024-03-19T03:07:25.570 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.294+0000 7f5a76f68700 1 -- 172.21.15.59:0/2938197409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a70114a50 con 0x7f5a70102720 2024-03-19T03:07:25.570 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.294+0000 7f5a6d7fa700 1 -- 172.21.15.59:0/2938197409 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f5a6401a4c0 con 0x7f5a70102720 2024-03-19T03:07:25.570 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.295+0000 7f5a6d7fa700 1 --2- 172.21.15.59:0/2938197409 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f5a600367e0 0x7f5a60038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:25.570 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.295+0000 7f5a6d7fa700 1 -- 172.21.15.59:0/2938197409 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5a6404a8b0 con 0x7f5a70102720 2024-03-19T03:07:25.570 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.295+0000 7f5a6ffff700 1 --2- 172.21.15.59:0/2938197409 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f5a600367e0 0x7f5a60038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:25.571 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.295+0000 7f5a76f68700 1 -- 172.21.15.59:0/2938197409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a7005f5f0 con 0x7f5a70102720 2024-03-19T03:07:25.571 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.295+0000 7f5a6ffff700 1 --2- 172.21.15.59:0/2938197409 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f5a600367e0 0x7f5a60038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c009990 tx=0x7f5a5c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:25.571 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.301+0000 7f5a6d7fa700 1 -- 172.21.15.59:0/2938197409 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f5a64018940 con 0x7f5a70102720 2024-03-19T03:07:25.571 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.516+0000 7f5a76f68700 1 -- 172.21.15.59:0/2938197409 --> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f5a7004bed0 con 0x7f5a600367e0 2024-03-19T03:07:25.571 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.763+0000 7f5a6d7fa700 1 -- 172.21.15.59:0/2938197409 <== mgr.14162 v2:172.21.15.59:6800/1857243897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f5a7004bed0 con 0x7f5a600367e0 2024-03-19T03:07:25.571 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.765+0000 7f5a76f68700 1 -- 172.21.15.59:0/2938197409 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f5a600367e0 msgr2=0x7f5a60038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:25.571 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.765+0000 7f5a76f68700 1 --2- 172.21.15.59:0/2938197409 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f5a600367e0 0x7f5a60038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c009990 tx=0x7f5a5c006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:25.571 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.765+0000 7f5a76f68700 1 -- 172.21.15.59:0/2938197409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a70102720 msgr2=0x7f5a701137a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:25.571 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.765+0000 7f5a76f68700 1 --2- 172.21.15.59:0/2938197409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a70102720 0x7f5a701137a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5a640090b0 tx=0x7f5a64009250 comp rx=0 tx=0).stop 2024-03-19T03:07:25.571 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.765+0000 7f5a76f68700 1 -- 172.21.15.59:0/2938197409 shutdown_connections 2024-03-19T03:07:25.572 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.765+0000 7f5a76f68700 1 --2- 172.21.15.59:0/2938197409 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f5a600367e0 0x7f5a60038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:25.572 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.765+0000 7f5a76f68700 1 --2- 172.21.15.59:0/2938197409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a70102720 0x7f5a701137a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:25.572 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.765+0000 7f5a76f68700 1 -- 172.21.15.59:0/2938197409 >> 172.21.15.59:0/2938197409 conn(0x7f5a700fde40 msgr2=0x7f5a700fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:25.572 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.765+0000 7f5a76f68700 1 -- 172.21.15.59:0/2938197409 shutdown_connections 2024-03-19T03:07:25.572 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:24.765+0000 7f5a76f68700 1 -- 172.21.15.59:0/2938197409 wait complete. 2024-03-19T03:07:25.572 INFO:teuthology.orchestra.run.smithi059.stdout:Fetching dashboard port number... 2024-03-19T03:07:26.275 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:26 smithi059 ceph-mon[29133]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:07:27.175 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:27 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/589827216' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-19T03:07:27.182 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 8443 2024-03-19T03:07:27.182 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.404+0000 7fd3d47a6700 1 Processor -- start 2024-03-19T03:07:27.182 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.404+0000 7fd3d47a6700 1 -- start start 2024-03-19T03:07:27.182 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.404+0000 7fd3d47a6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3cc077be0 0x7fd3cc0760f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:27.182 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.404+0000 7fd3d47a6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc077ff0 con 0x7fd3cc077be0 2024-03-19T03:07:27.182 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.404+0000 7fd3d2542700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3cc077be0 0x7fd3cc0760f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:27.182 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.404+0000 7fd3d2542700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3cc077be0 0x7fd3cc0760f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33050/0 (socket says 172.21.15.59:33050) 2024-03-19T03:07:27.182 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.404+0000 7fd3d2542700 1 -- 172.21.15.59:0/3791887928 learned_addr learned my addr 172.21.15.59:0/3791887928 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:27.182 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.405+0000 7fd3d2542700 1 -- 172.21.15.59:0/3791887928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3cc076630 con 0x7fd3cc077be0 2024-03-19T03:07:27.182 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.405+0000 7fd3d2542700 1 --2- 172.21.15.59:0/3791887928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3cc077be0 0x7fd3cc0760f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd3c80098e0 tx=0x7fd3c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bb52c801826b7d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:27.183 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.405+0000 7fd3d1540700 1 -- 172.21.15.59:0/3791887928 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3c800b860 con 0x7fd3cc077be0 2024-03-19T03:07:27.183 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.405+0000 7fd3d1540700 1 -- 172.21.15.59:0/3791887928 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3c800be90 con 0x7fd3cc077be0 2024-03-19T03:07:27.183 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.405+0000 7fd3d1540700 1 -- 172.21.15.59:0/3791887928 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3c80103c0 con 0x7fd3cc077be0 2024-03-19T03:07:27.183 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.405+0000 7fd3d47a6700 1 -- 172.21.15.59:0/3791887928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3cc077be0 msgr2=0x7fd3cc0760f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:27.183 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.405+0000 7fd3d47a6700 1 --2- 172.21.15.59:0/3791887928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3cc077be0 0x7fd3cc0760f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd3c80098e0 tx=0x7fd3c8009250 comp rx=0 tx=0).stop 2024-03-19T03:07:27.183 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.405+0000 7fd3d47a6700 1 -- 172.21.15.59:0/3791887928 shutdown_connections 2024-03-19T03:07:27.183 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.405+0000 7fd3d47a6700 1 --2- 172.21.15.59:0/3791887928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3cc077be0 0x7fd3cc0760f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:27.183 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.405+0000 7fd3d47a6700 1 -- 172.21.15.59:0/3791887928 >> 172.21.15.59:0/3791887928 conn(0x7fd3cc0fde70 msgr2=0x7fd3cc100280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:27.183 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.406+0000 7fd3d47a6700 1 -- 172.21.15.59:0/3791887928 shutdown_connections 2024-03-19T03:07:27.183 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.406+0000 7fd3d47a6700 1 -- 172.21.15.59:0/3791887928 wait complete. 2024-03-19T03:07:27.183 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.406+0000 7fd3d47a6700 1 Processor -- start 2024-03-19T03:07:27.183 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.406+0000 7fd3d47a6700 1 -- start start 2024-03-19T03:07:27.183 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.406+0000 7fd3d47a6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3cc077be0 0x7fd3cc1137d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:27.183 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.406+0000 7fd3d47a6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc113d10 con 0x7fd3cc077be0 2024-03-19T03:07:27.183 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.406+0000 7fd3d2542700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3cc077be0 0x7fd3cc1137d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:27.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.406+0000 7fd3d2542700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3cc077be0 0x7fd3cc1137d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33062/0 (socket says 172.21.15.59:33062) 2024-03-19T03:07:27.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.406+0000 7fd3d2542700 1 -- 172.21.15.59:0/589827216 learned_addr learned my addr 172.21.15.59:0/589827216 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:27.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.407+0000 7fd3d2542700 1 -- 172.21.15.59:0/589827216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c80094f0 con 0x7fd3cc077be0 2024-03-19T03:07:27.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.407+0000 7fd3d2542700 1 --2- 172.21.15.59:0/589827216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3cc077be0 0x7fd3cc1137d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd3c80094c0 tx=0x7fd3c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:27.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.407+0000 7fd3c2ffd700 1 -- 172.21.15.59:0/589827216 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3c8010890 con 0x7fd3cc077be0 2024-03-19T03:07:27.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.407+0000 7fd3c2ffd700 1 -- 172.21.15.59:0/589827216 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3c8010ec0 con 0x7fd3cc077be0 2024-03-19T03:07:27.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.407+0000 7fd3d47a6700 1 -- 172.21.15.59:0/589827216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3cc113f10 con 0x7fd3cc077be0 2024-03-19T03:07:27.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.407+0000 7fd3c2ffd700 1 -- 172.21.15.59:0/589827216 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3c80243f0 con 0x7fd3cc077be0 2024-03-19T03:07:27.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.407+0000 7fd3d47a6700 1 -- 172.21.15.59:0/589827216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3cc114a80 con 0x7fd3cc077be0 2024-03-19T03:07:27.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.408+0000 7fd3c2ffd700 1 -- 172.21.15.59:0/589827216 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fd3c80109f0 con 0x7fd3cc077be0 2024-03-19T03:07:27.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.408+0000 7fd3c2ffd700 1 --2- 172.21.15.59:0/589827216 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fd3b8036830 0x7fd3b8038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:27.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.408+0000 7fd3c2ffd700 1 -- 172.21.15.59:0/589827216 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd3c804a370 con 0x7fd3cc077be0 2024-03-19T03:07:27.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.408+0000 7fd3d1d41700 1 --2- 172.21.15.59:0/589827216 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fd3b8036830 0x7fd3b8038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:27.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.408+0000 7fd3d47a6700 1 -- 172.21.15.59:0/589827216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3cc05f5f0 con 0x7fd3cc077be0 2024-03-19T03:07:27.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.409+0000 7fd3d1d41700 1 --2- 172.21.15.59:0/589827216 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fd3b8036830 0x7fd3b8038ce0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc009990 tx=0x7fd3bc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:27.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.416+0000 7fd3c2ffd700 1 -- 172.21.15.59:0/589827216 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fd3c802b940 con 0x7fd3cc077be0 2024-03-19T03:07:27.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.610+0000 7fd3d47a6700 1 -- 172.21.15.59:0/589827216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fd3cc04bed0 con 0x7fd3cc077be0 2024-03-19T03:07:27.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.610+0000 7fd3c2ffd700 1 -- 172.21.15.59:0/589827216 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fd3c802b2f0 con 0x7fd3cc077be0 2024-03-19T03:07:27.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.612+0000 7fd3d47a6700 1 -- 172.21.15.59:0/589827216 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fd3b8036830 msgr2=0x7fd3b8038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:27.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.612+0000 7fd3d47a6700 1 --2- 172.21.15.59:0/589827216 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fd3b8036830 0x7fd3b8038ce0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc009990 tx=0x7fd3bc006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:27.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.612+0000 7fd3d47a6700 1 -- 172.21.15.59:0/589827216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3cc077be0 msgr2=0x7fd3cc1137d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:27.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.612+0000 7fd3d47a6700 1 --2- 172.21.15.59:0/589827216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3cc077be0 0x7fd3cc1137d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd3c80094c0 tx=0x7fd3c8009250 comp rx=0 tx=0).stop 2024-03-19T03:07:27.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.612+0000 7fd3d47a6700 1 -- 172.21.15.59:0/589827216 shutdown_connections 2024-03-19T03:07:27.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.612+0000 7fd3d47a6700 1 --2- 172.21.15.59:0/589827216 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fd3b8036830 0x7fd3b8038ce0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:27.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.612+0000 7fd3d47a6700 1 --2- 172.21.15.59:0/589827216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3cc077be0 0x7fd3cc1137d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:27.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.612+0000 7fd3d47a6700 1 -- 172.21.15.59:0/589827216 >> 172.21.15.59:0/589827216 conn(0x7fd3cc0fde70 msgr2=0x7fd3cc0fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:27.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.612+0000 7fd3d47a6700 1 -- 172.21.15.59:0/589827216 shutdown_connections 2024-03-19T03:07:27.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:26.612+0000 7fd3d47a6700 1 -- 172.21.15.59:0/589827216 wait complete. 2024-03-19T03:07:27.189 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-19T03:07:27.189 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout disabled 2024-03-19T03:07:27.196 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-19T03:07:27.196 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout inactive 2024-03-19T03:07:27.196 INFO:teuthology.orchestra.run.smithi059.stdout:firewalld.service is not enabled 2024-03-19T03:07:27.196 INFO:teuthology.orchestra.run.smithi059.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-19T03:07:27.198 INFO:teuthology.orchestra.run.smithi059.stdout:Ceph Dashboard is now available at: 2024-03-19T03:07:27.198 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:07:27.198 INFO:teuthology.orchestra.run.smithi059.stdout: URL: https://smithi059.front.sepia.ceph.com:8443/ 2024-03-19T03:07:27.198 INFO:teuthology.orchestra.run.smithi059.stdout: User: admin 2024-03-19T03:07:27.199 INFO:teuthology.orchestra.run.smithi059.stdout: Password: 704a0l3u8y 2024-03-19T03:07:27.199 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:07:27.199 INFO:teuthology.orchestra.run.smithi059.stdout:Saving cluster configuration to /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/config directory 2024-03-19T03:07:27.199 INFO:teuthology.orchestra.run.smithi059.stdout:Enabling autotune for osd_memory_target 2024-03-19T03:07:28.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:28 smithi059 ceph-mon[29133]: mgrmap e13: smithi059.azpecc(active, since 6s) 2024-03-19T03:07:28.280 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.604+0000 7f8cfdd9b700 1 Processor -- start 2024-03-19T03:07:28.280 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.604+0000 7f8cfdd9b700 1 -- start start 2024-03-19T03:07:28.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.604+0000 7f8cfdd9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cf8076b40 0x7f8cf81053b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:28.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.604+0000 7f8cfdd9b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf81058f0 con 0x7f8cf8076b40 2024-03-19T03:07:28.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.605+0000 7f8cf77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cf8076b40 0x7f8cf81053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:28.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.605+0000 7f8cf77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cf8076b40 0x7f8cf81053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33072/0 (socket says 172.21.15.59:33072) 2024-03-19T03:07:28.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.605+0000 7f8cf77fe700 1 -- 172.21.15.59:0/2178636019 learned_addr learned my addr 172.21.15.59:0/2178636019 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:28.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.605+0000 7f8cf77fe700 1 -- 172.21.15.59:0/2178636019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf8105a30 con 0x7f8cf8076b40 2024-03-19T03:07:28.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.605+0000 7f8cf77fe700 1 --2- 172.21.15.59:0/2178636019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cf8076b40 0x7f8cf81053b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8ce80098e0 tx=0x7f8ce8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e322308c28675a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:28.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.605+0000 7f8cf67fc700 1 -- 172.21.15.59:0/2178636019 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ce800b860 con 0x7f8cf8076b40 2024-03-19T03:07:28.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.605+0000 7f8cf67fc700 1 -- 172.21.15.59:0/2178636019 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ce800be90 con 0x7f8cf8076b40 2024-03-19T03:07:28.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.605+0000 7f8cf67fc700 1 -- 172.21.15.59:0/2178636019 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ce80103c0 con 0x7f8cf8076b40 2024-03-19T03:07:28.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.606+0000 7f8cfdd9b700 1 -- 172.21.15.59:0/2178636019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cf8076b40 msgr2=0x7f8cf81053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:28.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.606+0000 7f8cfdd9b700 1 --2- 172.21.15.59:0/2178636019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cf8076b40 0x7f8cf81053b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8ce80098e0 tx=0x7f8ce8009250 comp rx=0 tx=0).stop 2024-03-19T03:07:28.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.606+0000 7f8cfdd9b700 1 -- 172.21.15.59:0/2178636019 shutdown_connections 2024-03-19T03:07:28.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.606+0000 7f8cfdd9b700 1 --2- 172.21.15.59:0/2178636019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cf8076b40 0x7f8cf81053b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:28.282 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.606+0000 7f8cfdd9b700 1 -- 172.21.15.59:0/2178636019 >> 172.21.15.59:0/2178636019 conn(0x7f8cf80fcdc0 msgr2=0x7f8cf80ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:28.282 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.606+0000 7f8cfdd9b700 1 -- 172.21.15.59:0/2178636019 shutdown_connections 2024-03-19T03:07:28.282 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.606+0000 7f8cfdd9b700 1 -- 172.21.15.59:0/2178636019 wait complete. 2024-03-19T03:07:28.282 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.606+0000 7f8cfdd9b700 1 Processor -- start 2024-03-19T03:07:28.282 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.606+0000 7f8cfdd9b700 1 -- start start 2024-03-19T03:07:28.282 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.607+0000 7f8cfdd9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cf8076b40 0x7f8cf8111590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:28.282 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.607+0000 7f8cfdd9b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf8111ad0 con 0x7f8cf8076b40 2024-03-19T03:07:28.282 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.607+0000 7f8cf77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cf8076b40 0x7f8cf8111590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:28.282 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.607+0000 7f8cf77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cf8076b40 0x7f8cf8111590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33088/0 (socket says 172.21.15.59:33088) 2024-03-19T03:07:28.282 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.607+0000 7f8cf77fe700 1 -- 172.21.15.59:0/1226189301 learned_addr learned my addr 172.21.15.59:0/1226189301 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:28.282 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.607+0000 7f8cf77fe700 1 -- 172.21.15.59:0/1226189301 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce80094f0 con 0x7f8cf8076b40 2024-03-19T03:07:28.282 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.607+0000 7f8cf77fe700 1 --2- 172.21.15.59:0/1226189301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cf8076b40 0x7f8cf8111590 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8006d40 tx=0x7f8ce8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:28.282 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.607+0000 7f8cdffff700 1 -- 172.21.15.59:0/1226189301 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ce8010840 con 0x7f8cf8076b40 2024-03-19T03:07:28.283 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.607+0000 7f8cfdd9b700 1 -- 172.21.15.59:0/1226189301 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cf8111cd0 con 0x7f8cf8076b40 2024-03-19T03:07:28.283 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.607+0000 7f8cdffff700 1 -- 172.21.15.59:0/1226189301 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ce8010e70 con 0x7f8cf8076b40 2024-03-19T03:07:28.283 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.607+0000 7f8cdffff700 1 -- 172.21.15.59:0/1226189301 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ce8024ea0 con 0x7f8cf8076b40 2024-03-19T03:07:28.283 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.607+0000 7f8cfdd9b700 1 -- 172.21.15.59:0/1226189301 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cf8112840 con 0x7f8cf8076b40 2024-03-19T03:07:28.283 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.608+0000 7f8cdffff700 1 -- 172.21.15.59:0/1226189301 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f8ce8024780 con 0x7f8cf8076b40 2024-03-19T03:07:28.283 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.608+0000 7f8cdffff700 1 --2- 172.21.15.59:0/1226189301 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f8ce0036830 0x7f8ce0038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:28.283 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.608+0000 7f8cfdd9b700 1 -- 172.21.15.59:0/1226189301 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cf805f5f0 con 0x7f8cf8076b40 2024-03-19T03:07:28.283 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.608+0000 7f8cdffff700 1 -- 172.21.15.59:0/1226189301 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8ce804a9d0 con 0x7f8cf8076b40 2024-03-19T03:07:28.283 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.608+0000 7f8cf6ffd700 1 --2- 172.21.15.59:0/1226189301 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f8ce0036830 0x7f8ce0038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:28.283 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.609+0000 7f8cf6ffd700 1 --2- 172.21.15.59:0/1226189301 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f8ce0036830 0x7f8ce0038ce0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8cec009990 tx=0x7f8cec006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:28.283 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.613+0000 7f8cdffff700 1 -- 172.21.15.59:0/1226189301 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f8ce802c520 con 0x7f8cf8076b40 2024-03-19T03:07:28.283 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.802+0000 7f8cfdd9b700 1 -- 172.21.15.59:0/1226189301 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f8cf8106760 con 0x7f8cf8076b40 2024-03-19T03:07:28.283 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.802+0000 7f8cdffff700 1 -- 172.21.15.59:0/1226189301 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f8ce802c340 con 0x7f8cf8076b40 2024-03-19T03:07:28.283 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.804+0000 7f8cfdd9b700 1 -- 172.21.15.59:0/1226189301 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f8ce0036830 msgr2=0x7f8ce0038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:28.284 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.804+0000 7f8cfdd9b700 1 --2- 172.21.15.59:0/1226189301 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f8ce0036830 0x7f8ce0038ce0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8cec009990 tx=0x7f8cec006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:28.284 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.804+0000 7f8cfdd9b700 1 -- 172.21.15.59:0/1226189301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cf8076b40 msgr2=0x7f8cf8111590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:28.285 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.804+0000 7f8cfdd9b700 1 --2- 172.21.15.59:0/1226189301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cf8076b40 0x7f8cf8111590 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8006d40 tx=0x7f8ce8009250 comp rx=0 tx=0).stop 2024-03-19T03:07:28.285 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.804+0000 7f8cfdd9b700 1 -- 172.21.15.59:0/1226189301 shutdown_connections 2024-03-19T03:07:28.285 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.804+0000 7f8cfdd9b700 1 --2- 172.21.15.59:0/1226189301 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f8ce0036830 0x7f8ce0038ce0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:28.285 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.804+0000 7f8cfdd9b700 1 --2- 172.21.15.59:0/1226189301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cf8076b40 0x7f8cf8111590 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:28.285 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.804+0000 7f8cfdd9b700 1 -- 172.21.15.59:0/1226189301 >> 172.21.15.59:0/1226189301 conn(0x7f8cf80fcdc0 msgr2=0x7f8cf80fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:28.285 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.804+0000 7f8cfdd9b700 1 -- 172.21.15.59:0/1226189301 shutdown_connections 2024-03-19T03:07:28.285 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:27.804+0000 7f8cfdd9b700 1 -- 172.21.15.59:0/1226189301 wait complete. 2024-03-19T03:07:29.579 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.708+0000 7f9e69abf700 1 Processor -- start 2024-03-19T03:07:29.580 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.708+0000 7f9e69abf700 1 -- start start 2024-03-19T03:07:29.580 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.708+0000 7f9e69abf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e64103d00 0x7f9e64104110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:29.580 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.708+0000 7f9e69abf700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e64104650 con 0x7f9e64103d00 2024-03-19T03:07:29.580 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.708+0000 7f9e637fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e64103d00 0x7f9e64104110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:29.580 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.708+0000 7f9e637fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e64103d00 0x7f9e64104110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33096/0 (socket says 172.21.15.59:33096) 2024-03-19T03:07:29.581 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.708+0000 7f9e637fe700 1 -- 172.21.15.59:0/4191692384 learned_addr learned my addr 172.21.15.59:0/4191692384 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:29.581 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.708+0000 7f9e637fe700 1 -- 172.21.15.59:0/4191692384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e64104790 con 0x7f9e64103d00 2024-03-19T03:07:29.581 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.709+0000 7f9e637fe700 1 --2- 172.21.15.59:0/4191692384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e64103d00 0x7f9e64104110 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9e540098e0 tx=0x7f9e54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c74b5237a6db8775 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:29.581 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.709+0000 7f9e627fc700 1 -- 172.21.15.59:0/4191692384 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e5400b860 con 0x7f9e64103d00 2024-03-19T03:07:29.582 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.709+0000 7f9e627fc700 1 -- 172.21.15.59:0/4191692384 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e5400be90 con 0x7f9e64103d00 2024-03-19T03:07:29.582 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.709+0000 7f9e627fc700 1 -- 172.21.15.59:0/4191692384 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e540103c0 con 0x7f9e64103d00 2024-03-19T03:07:29.582 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.709+0000 7f9e69abf700 1 -- 172.21.15.59:0/4191692384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e64103d00 msgr2=0x7f9e64104110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:29.582 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.709+0000 7f9e69abf700 1 --2- 172.21.15.59:0/4191692384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e64103d00 0x7f9e64104110 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9e540098e0 tx=0x7f9e54009250 comp rx=0 tx=0).stop 2024-03-19T03:07:29.582 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.709+0000 7f9e69abf700 1 -- 172.21.15.59:0/4191692384 shutdown_connections 2024-03-19T03:07:29.583 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.709+0000 7f9e69abf700 1 --2- 172.21.15.59:0/4191692384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e64103d00 0x7f9e64104110 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:29.583 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.709+0000 7f9e69abf700 1 -- 172.21.15.59:0/4191692384 >> 172.21.15.59:0/4191692384 conn(0x7f9e640fc500 msgr2=0x7f9e640fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:29.583 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.709+0000 7f9e69abf700 1 -- 172.21.15.59:0/4191692384 shutdown_connections 2024-03-19T03:07:29.583 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.709+0000 7f9e69abf700 1 -- 172.21.15.59:0/4191692384 wait complete. 2024-03-19T03:07:29.583 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.710+0000 7f9e69abf700 1 Processor -- start 2024-03-19T03:07:29.583 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.710+0000 7f9e69abf700 1 -- start start 2024-03-19T03:07:29.584 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.710+0000 7f9e69abf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e64103d00 0x7f9e64111590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:29.584 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.710+0000 7f9e69abf700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e64111ad0 con 0x7f9e64103d00 2024-03-19T03:07:29.584 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.710+0000 7f9e637fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e64103d00 0x7f9e64111590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:29.584 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.710+0000 7f9e637fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e64103d00 0x7f9e64111590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33110/0 (socket says 172.21.15.59:33110) 2024-03-19T03:07:29.584 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.710+0000 7f9e637fe700 1 -- 172.21.15.59:0/617451115 learned_addr learned my addr 172.21.15.59:0/617451115 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:29.584 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.711+0000 7f9e637fe700 1 -- 172.21.15.59:0/617451115 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e540094f0 con 0x7f9e64103d00 2024-03-19T03:07:29.585 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.711+0000 7f9e637fe700 1 --2- 172.21.15.59:0/617451115 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e64103d00 0x7f9e64111590 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9e54011680 tx=0x7f9e54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:29.585 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.711+0000 7f9e4bfff700 1 -- 172.21.15.59:0/617451115 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e54010890 con 0x7f9e64103d00 2024-03-19T03:07:29.585 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.711+0000 7f9e4bfff700 1 -- 172.21.15.59:0/617451115 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e54010ec0 con 0x7f9e64103d00 2024-03-19T03:07:29.585 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.711+0000 7f9e4bfff700 1 -- 172.21.15.59:0/617451115 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e540243f0 con 0x7f9e64103d00 2024-03-19T03:07:29.585 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.711+0000 7f9e69abf700 1 -- 172.21.15.59:0/617451115 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e64111cd0 con 0x7f9e64103d00 2024-03-19T03:07:29.585 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.711+0000 7f9e69abf700 1 -- 172.21.15.59:0/617451115 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e64112840 con 0x7f9e64103d00 2024-03-19T03:07:29.586 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.712+0000 7f9e4bfff700 1 -- 172.21.15.59:0/617451115 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f9e540109f0 con 0x7f9e64103d00 2024-03-19T03:07:29.586 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.712+0000 7f9e69abf700 1 -- 172.21.15.59:0/617451115 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e6405f5f0 con 0x7f9e64103d00 2024-03-19T03:07:29.586 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.712+0000 7f9e4bfff700 1 --2- 172.21.15.59:0/617451115 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f9e4c0368d0 0x7f9e4c038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:29.586 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.712+0000 7f9e4bfff700 1 -- 172.21.15.59:0/617451115 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9e5404a3f0 con 0x7f9e64103d00 2024-03-19T03:07:29.587 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.712+0000 7f9e62ffd700 1 --2- 172.21.15.59:0/617451115 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f9e4c0368d0 0x7f9e4c038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:29.587 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.713+0000 7f9e62ffd700 1 --2- 172.21.15.59:0/617451115 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f9e4c0368d0 0x7f9e4c038d80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9e58009990 tx=0x7f9e58006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:29.587 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:28.717+0000 7f9e4bfff700 1 -- 172.21.15.59:0/617451115 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f9e5402b8c0 con 0x7f9e64103d00 2024-03-19T03:07:29.587 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:29.017+0000 7f9e69abf700 1 -- 172.21.15.59:0/617451115 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f9e6404bed0 con 0x7f9e64103d00 2024-03-19T03:07:29.587 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:29.020+0000 7f9e4bfff700 1 -- 172.21.15.59:0/617451115 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f9e5402b2f0 con 0x7f9e64103d00 2024-03-19T03:07:29.587 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:29.022+0000 7f9e69abf700 1 -- 172.21.15.59:0/617451115 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f9e4c0368d0 msgr2=0x7f9e4c038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:29.588 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:29.022+0000 7f9e69abf700 1 --2- 172.21.15.59:0/617451115 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f9e4c0368d0 0x7f9e4c038d80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9e58009990 tx=0x7f9e58006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:29.588 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:29.022+0000 7f9e69abf700 1 -- 172.21.15.59:0/617451115 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e64103d00 msgr2=0x7f9e64111590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:29.588 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:29.022+0000 7f9e69abf700 1 --2- 172.21.15.59:0/617451115 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e64103d00 0x7f9e64111590 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9e54011680 tx=0x7f9e54009250 comp rx=0 tx=0).stop 2024-03-19T03:07:29.588 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:29.022+0000 7f9e69abf700 1 -- 172.21.15.59:0/617451115 shutdown_connections 2024-03-19T03:07:29.588 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:29.022+0000 7f9e69abf700 1 --2- 172.21.15.59:0/617451115 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f9e4c0368d0 0x7f9e4c038d80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:29.588 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:29.022+0000 7f9e69abf700 1 --2- 172.21.15.59:0/617451115 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e64103d00 0x7f9e64111590 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:29.588 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:29.022+0000 7f9e69abf700 1 -- 172.21.15.59:0/617451115 >> 172.21.15.59:0/617451115 conn(0x7f9e640fc500 msgr2=0x7f9e640fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:29.589 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:29.023+0000 7f9e69abf700 1 -- 172.21.15.59:0/617451115 shutdown_connections 2024-03-19T03:07:29.589 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-03-19T03:07:29.023+0000 7f9e69abf700 1 -- 172.21.15.59:0/617451115 wait complete. 2024-03-19T03:07:29.589 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-19T03:07:29.589 INFO:teuthology.orchestra.run.smithi059.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-19T03:07:29.589 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:07:29.589 INFO:teuthology.orchestra.run.smithi059.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-19T03:07:29.590 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:07:29.590 INFO:teuthology.orchestra.run.smithi059.stdout:Or, if you are only running a single cluster on this host: 2024-03-19T03:07:29.590 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:07:29.590 INFO:teuthology.orchestra.run.smithi059.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-19T03:07:29.590 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:07:29.590 INFO:teuthology.orchestra.run.smithi059.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-19T03:07:29.591 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:07:29.591 INFO:teuthology.orchestra.run.smithi059.stdout: ceph telemetry on 2024-03-19T03:07:29.591 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:07:29.591 INFO:teuthology.orchestra.run.smithi059.stdout:For more information see: 2024-03-19T03:07:29.591 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:07:29.591 INFO:teuthology.orchestra.run.smithi059.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-03-19T03:07:29.592 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:07:29.592 INFO:teuthology.orchestra.run.smithi059.stdout:Bootstrap complete. 2024-03-19T03:07:29.612 INFO:tasks.cephadm:Fetching config... 2024-03-19T03:07:29.612 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-19T03:07:29.613 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-19T03:07:29.668 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-19T03:07:29.668 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-19T03:07:29.668 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-19T03:07:29.723 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-19T03:07:29.723 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-19T03:07:29.723 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/keyring of=/dev/stdout 2024-03-19T03:07:29.790 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-19T03:07:29.790 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-19T03:07:29.791 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-19T03:07:29.844 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-19T03:07:29.845 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOlnUuONg0bL3nPOacT9ojXYWnGrvA7ywhiXYrtKhCOhO4O/V1DrisuAIKAWo4HBCZAwlm2i+RzoqZiuxgXWRVwOFfV5s4vTEtCRr3b0gx61HfJHz5tatZwKSRbSxNYA63NEdDfRczfboTZFXFOvYoX4680X15HU3I1oYgU84yJ6raLH0AzELJiMrmKfWEe4dszmwgRtzW/qNXETin8TTKlbyj8s2QEwwyJw4YRRpXQfxyit1NpdkOvSybiXhbHGSkX9CK2DSdZi4uz9XrQ9aHsaUrUgRXoZ6MMv6T8HD0tGEDs2GMAfhwCMFEPn3ydTEANZ1dkp9SwhpzqM6widkBMSXinOYICxyJcn+BLYex3+lMmfqN90+4EzoFab7j3aJpptof5uNTbR1XnTx3GVw3wdvFaFLSNuLlviciPpqzjHVbAOCJ3EU2ilGaFHBMKLFaaGcc20iAtKx87Ot0DP1XjEmB6j1brlVI/k1zerfVyMUf+oVutj65x+TzzkJbGnE= ceph-8c8ae530-e59d-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-19T03:07:29.924 INFO:teuthology.orchestra.run.smithi059.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOlnUuONg0bL3nPOacT9ojXYWnGrvA7ywhiXYrtKhCOhO4O/V1DrisuAIKAWo4HBCZAwlm2i+RzoqZiuxgXWRVwOFfV5s4vTEtCRr3b0gx61HfJHz5tatZwKSRbSxNYA63NEdDfRczfboTZFXFOvYoX4680X15HU3I1oYgU84yJ6raLH0AzELJiMrmKfWEe4dszmwgRtzW/qNXETin8TTKlbyj8s2QEwwyJw4YRRpXQfxyit1NpdkOvSybiXhbHGSkX9CK2DSdZi4uz9XrQ9aHsaUrUgRXoZ6MMv6T8HD0tGEDs2GMAfhwCMFEPn3ydTEANZ1dkp9SwhpzqM6widkBMSXinOYICxyJcn+BLYex3+lMmfqN90+4EzoFab7j3aJpptof5uNTbR1XnTx3GVw3wdvFaFLSNuLlviciPpqzjHVbAOCJ3EU2ilGaFHBMKLFaaGcc20iAtKx87Ot0DP1XjEmB6j1brlVI/k1zerfVyMUf+oVutj65x+TzzkJbGnE= ceph-8c8ae530-e59d-11ee-95c9-87774f69a715 2024-03-19T03:07:29.939 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOlnUuONg0bL3nPOacT9ojXYWnGrvA7ywhiXYrtKhCOhO4O/V1DrisuAIKAWo4HBCZAwlm2i+RzoqZiuxgXWRVwOFfV5s4vTEtCRr3b0gx61HfJHz5tatZwKSRbSxNYA63NEdDfRczfboTZFXFOvYoX4680X15HU3I1oYgU84yJ6raLH0AzELJiMrmKfWEe4dszmwgRtzW/qNXETin8TTKlbyj8s2QEwwyJw4YRRpXQfxyit1NpdkOvSybiXhbHGSkX9CK2DSdZi4uz9XrQ9aHsaUrUgRXoZ6MMv6T8HD0tGEDs2GMAfhwCMFEPn3ydTEANZ1dkp9SwhpzqM6widkBMSXinOYICxyJcn+BLYex3+lMmfqN90+4EzoFab7j3aJpptof5uNTbR1XnTx3GVw3wdvFaFLSNuLlviciPpqzjHVbAOCJ3EU2ilGaFHBMKLFaaGcc20iAtKx87Ot0DP1XjEmB6j1brlVI/k1zerfVyMUf+oVutj65x+TzzkJbGnE= ceph-8c8ae530-e59d-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-19T03:07:29.983 INFO:teuthology.orchestra.run.smithi176.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOlnUuONg0bL3nPOacT9ojXYWnGrvA7ywhiXYrtKhCOhO4O/V1DrisuAIKAWo4HBCZAwlm2i+RzoqZiuxgXWRVwOFfV5s4vTEtCRr3b0gx61HfJHz5tatZwKSRbSxNYA63NEdDfRczfboTZFXFOvYoX4680X15HU3I1oYgU84yJ6raLH0AzELJiMrmKfWEe4dszmwgRtzW/qNXETin8TTKlbyj8s2QEwwyJw4YRRpXQfxyit1NpdkOvSybiXhbHGSkX9CK2DSdZi4uz9XrQ9aHsaUrUgRXoZ6MMv6T8HD0tGEDs2GMAfhwCMFEPn3ydTEANZ1dkp9SwhpzqM6widkBMSXinOYICxyJcn+BLYex3+lMmfqN90+4EzoFab7j3aJpptof5uNTbR1XnTx3GVw3wdvFaFLSNuLlviciPpqzjHVbAOCJ3EU2ilGaFHBMKLFaaGcc20iAtKx87Ot0DP1XjEmB6j1brlVI/k1zerfVyMUf+oVutj65x+TzzkJbGnE= ceph-8c8ae530-e59d-11ee-95c9-87774f69a715 2024-03-19T03:07:29.998 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-19T03:07:30.260 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:07:30.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:30 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/617451115' entity='client.admin' 2024-03-19T03:07:32.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.557+0000 7fc568bfa700 1 -- 172.21.15.59:0/1188822619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5640fb960 msgr2=0x7fc5640fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:32.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.557+0000 7fc568bfa700 1 --2- 172.21.15.59:0/1188822619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5640fb960 0x7fc5640fbd70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc554009c60 tx=0x7fc554004d10 comp rx=0 tx=0).stop 2024-03-19T03:07:32.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.558+0000 7fc568bfa700 1 -- 172.21.15.59:0/1188822619 shutdown_connections 2024-03-19T03:07:32.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.558+0000 7fc568bfa700 1 --2- 172.21.15.59:0/1188822619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5640fb960 0x7fc5640fbd70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:32.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.558+0000 7fc568bfa700 1 -- 172.21.15.59:0/1188822619 >> 172.21.15.59:0/1188822619 conn(0x7fc5640f70c0 msgr2=0x7fc5640f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:32.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.558+0000 7fc568bfa700 1 -- 172.21.15.59:0/1188822619 shutdown_connections 2024-03-19T03:07:32.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.558+0000 7fc568bfa700 1 -- 172.21.15.59:0/1188822619 wait complete. 2024-03-19T03:07:32.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.558+0000 7fc568bfa700 1 Processor -- start 2024-03-19T03:07:32.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.558+0000 7fc568bfa700 1 -- start start 2024-03-19T03:07:32.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.559+0000 7fc568bfa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5640fb960 0x7fc56410bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:32.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.559+0000 7fc568bfa700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56410c500 con 0x7fc5640fb960 2024-03-19T03:07:32.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.559+0000 7fc56259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5640fb960 0x7fc56410bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:32.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.559+0000 7fc56259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5640fb960 0x7fc56410bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33126/0 (socket says 172.21.15.59:33126) 2024-03-19T03:07:32.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.559+0000 7fc56259c700 1 -- 172.21.15.59:0/3711191909 learned_addr learned my addr 172.21.15.59:0/3711191909 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:32.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.559+0000 7fc56259c700 1 -- 172.21.15.59:0/3711191909 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc554009940 con 0x7fc5640fb960 2024-03-19T03:07:32.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.559+0000 7fc56259c700 1 --2- 172.21.15.59:0/3711191909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5640fb960 0x7fc56410bfc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc554009500 tx=0x7fc554004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:32.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.559+0000 7fc55affd700 1 -- 172.21.15.59:0/3711191909 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc554007c80 con 0x7fc5640fb960 2024-03-19T03:07:32.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.559+0000 7fc568bfa700 1 -- 172.21.15.59:0/3711191909 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc56410c700 con 0x7fc5640fb960 2024-03-19T03:07:32.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.559+0000 7fc55affd700 1 -- 172.21.15.59:0/3711191909 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc5540198f0 con 0x7fc5640fb960 2024-03-19T03:07:32.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.559+0000 7fc568bfa700 1 -- 172.21.15.59:0/3711191909 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc56410d270 con 0x7fc5640fb960 2024-03-19T03:07:32.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.559+0000 7fc55affd700 1 -- 172.21.15.59:0/3711191909 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5540129b0 con 0x7fc5640fb960 2024-03-19T03:07:32.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.560+0000 7fc55affd700 1 -- 172.21.15.59:0/3711191909 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fc554012b10 con 0x7fc5640fb960 2024-03-19T03:07:32.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.560+0000 7fc55affd700 1 --2- 172.21.15.59:0/3711191909 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fc5500368d0 0x7fc550038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:32.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.560+0000 7fc55affd700 1 -- 172.21.15.59:0/3711191909 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc55404b840 con 0x7fc5640fb960 2024-03-19T03:07:32.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.560+0000 7fc568bfa700 1 -- 172.21.15.59:0/3711191909 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc564101e70 con 0x7fc5640fb960 2024-03-19T03:07:32.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.560+0000 7fc561d9b700 1 --2- 172.21.15.59:0/3711191909 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fc5500368d0 0x7fc550038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:32.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.561+0000 7fc561d9b700 1 --2- 172.21.15.59:0/3711191909 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fc5500368d0 0x7fc550038d80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc54c009990 tx=0x7fc54c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:32.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.565+0000 7fc55affd700 1 -- 172.21.15.59:0/3711191909 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fc554026080 con 0x7fc5640fb960 2024-03-19T03:07:32.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.757+0000 7fc568bfa700 1 -- 172.21.15.59:0/3711191909 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fc56405f5f0 con 0x7fc5640fb960 2024-03-19T03:07:32.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.761+0000 7fc55affd700 1 -- 172.21.15.59:0/3711191909 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fc554017030 con 0x7fc5640fb960 2024-03-19T03:07:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.763+0000 7fc568bfa700 1 -- 172.21.15.59:0/3711191909 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fc5500368d0 msgr2=0x7fc550038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.763+0000 7fc568bfa700 1 --2- 172.21.15.59:0/3711191909 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fc5500368d0 0x7fc550038d80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc54c009990 tx=0x7fc54c006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.763+0000 7fc568bfa700 1 -- 172.21.15.59:0/3711191909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5640fb960 msgr2=0x7fc56410bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.763+0000 7fc568bfa700 1 --2- 172.21.15.59:0/3711191909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5640fb960 0x7fc56410bfc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc554009500 tx=0x7fc554004d10 comp rx=0 tx=0).stop 2024-03-19T03:07:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.763+0000 7fc568bfa700 1 -- 172.21.15.59:0/3711191909 shutdown_connections 2024-03-19T03:07:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.763+0000 7fc568bfa700 1 --2- 172.21.15.59:0/3711191909 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fc5500368d0 0x7fc550038d80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.763+0000 7fc568bfa700 1 --2- 172.21.15.59:0/3711191909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5640fb960 0x7fc56410bfc0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.763+0000 7fc568bfa700 1 -- 172.21.15.59:0/3711191909 >> 172.21.15.59:0/3711191909 conn(0x7fc5640f70c0 msgr2=0x7fc564102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:32.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.763+0000 7fc568bfa700 1 -- 172.21.15.59:0/3711191909 shutdown_connections 2024-03-19T03:07:32.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:32.763+0000 7fc568bfa700 1 -- 172.21.15.59:0/3711191909 wait complete. 2024-03-19T03:07:33.086 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:32 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:33.086 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:32 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:33.086 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:32 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd/host:smithi059", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:07:33.086 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:32 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:33.086 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:32 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi059", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-19T03:07:33.086 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:32 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi059", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-19T03:07:33.086 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:32 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:07:33.086 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:32 smithi059 ceph-mon[29133]: Deploying daemon crash.smithi059 on smithi059 2024-03-19T03:07:33.087 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:32 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/3711191909' entity='client.admin' 2024-03-19T03:07:33.385 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-19T03:07:33.385 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-19T03:07:34.006 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:07:35.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.666+0000 7f905deb0700 1 -- 172.21.15.59:0/4130753457 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90580fb950 msgr2=0x7f90580fbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:35.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.666+0000 7f905deb0700 1 --2- 172.21.15.59:0/4130753457 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90580fb950 0x7f90580fbd60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f90480048c0 tx=0x7f9048009e30 comp rx=0 tx=0).stop 2024-03-19T03:07:35.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.666+0000 7f905deb0700 1 -- 172.21.15.59:0/4130753457 shutdown_connections 2024-03-19T03:07:35.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.666+0000 7f905deb0700 1 --2- 172.21.15.59:0/4130753457 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90580fb950 0x7f90580fbd60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:35.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.666+0000 7f905deb0700 1 -- 172.21.15.59:0/4130753457 >> 172.21.15.59:0/4130753457 conn(0x7f90580f7090 msgr2=0x7f90580f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:35.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.666+0000 7f905deb0700 1 -- 172.21.15.59:0/4130753457 shutdown_connections 2024-03-19T03:07:35.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.666+0000 7f905deb0700 1 -- 172.21.15.59:0/4130753457 wait complete. 2024-03-19T03:07:35.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.666+0000 7f905deb0700 1 Processor -- start 2024-03-19T03:07:35.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.666+0000 7f905deb0700 1 -- start start 2024-03-19T03:07:35.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.666+0000 7f905deb0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90580fb950 0x7f905810bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:35.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.666+0000 7f905deb0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f905810c4f0 con 0x7f90580fb950 2024-03-19T03:07:35.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.667+0000 7f90577fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90580fb950 0x7f905810bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:35.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.667+0000 7f90577fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90580fb950 0x7f905810bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56024/0 (socket says 172.21.15.59:56024) 2024-03-19T03:07:35.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.667+0000 7f90577fe700 1 -- 172.21.15.59:0/3211551228 learned_addr learned my addr 172.21.15.59:0/3211551228 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:35.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.667+0000 7f90577fe700 1 -- 172.21.15.59:0/3211551228 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9048005040 con 0x7f90580fb950 2024-03-19T03:07:35.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.667+0000 7f90577fe700 1 --2- 172.21.15.59:0/3211551228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90580fb950 0x7f905810bfb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f90480096b0 tx=0x7f9048009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:35.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.667+0000 7f903ffff700 1 -- 172.21.15.59:0/3211551228 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f904800bed0 con 0x7f90580fb950 2024-03-19T03:07:35.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.667+0000 7f903ffff700 1 -- 172.21.15.59:0/3211551228 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f904800b5c0 con 0x7f90580fb950 2024-03-19T03:07:35.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.667+0000 7f905deb0700 1 -- 172.21.15.59:0/3211551228 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f905810c6f0 con 0x7f90580fb950 2024-03-19T03:07:35.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.667+0000 7f903ffff700 1 -- 172.21.15.59:0/3211551228 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f904801c5b0 con 0x7f90580fb950 2024-03-19T03:07:35.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.667+0000 7f905deb0700 1 -- 172.21.15.59:0/3211551228 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f905810d260 con 0x7f90580fb950 2024-03-19T03:07:35.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.668+0000 7f903ffff700 1 -- 172.21.15.59:0/3211551228 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f9048013420 con 0x7f90580fb950 2024-03-19T03:07:35.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.668+0000 7f903ffff700 1 --2- 172.21.15.59:0/3211551228 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f9040036880 0x7f9040038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:35.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.668+0000 7f905deb0700 1 -- 172.21.15.59:0/3211551228 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9058101e60 con 0x7f90580fb950 2024-03-19T03:07:35.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.668+0000 7f903ffff700 1 -- 172.21.15.59:0/3211551228 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f90480499c0 con 0x7f90580fb950 2024-03-19T03:07:35.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.668+0000 7f9056ffd700 1 --2- 172.21.15.59:0/3211551228 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f9040036880 0x7f9040038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:35.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.669+0000 7f9056ffd700 1 --2- 172.21.15.59:0/3211551228 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f9040036880 0x7f9040038d30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f904c009990 tx=0x7f904c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:35.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.673+0000 7f903ffff700 1 -- 172.21.15.59:0/3211551228 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f9048011550 con 0x7f90580fb950 2024-03-19T03:07:35.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.878+0000 7f905deb0700 1 -- 172.21.15.59:0/3211551228 --> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f905810d4a0 con 0x7f9040036880 2024-03-19T03:07:35.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.881+0000 7f903ffff700 1 -- 172.21.15.59:0/3211551228 <== mgr.14162 v2:172.21.15.59:6800/1857243897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f905810d4a0 con 0x7f9040036880 2024-03-19T03:07:35.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.883+0000 7f905deb0700 1 -- 172.21.15.59:0/3211551228 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f9040036880 msgr2=0x7f9040038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:35.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.883+0000 7f905deb0700 1 --2- 172.21.15.59:0/3211551228 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f9040036880 0x7f9040038d30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f904c009990 tx=0x7f904c006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:35.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.883+0000 7f905deb0700 1 -- 172.21.15.59:0/3211551228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90580fb950 msgr2=0x7f905810bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:35.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.883+0000 7f905deb0700 1 --2- 172.21.15.59:0/3211551228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90580fb950 0x7f905810bfb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f90480096b0 tx=0x7f9048009e30 comp rx=0 tx=0).stop 2024-03-19T03:07:35.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.883+0000 7f905deb0700 1 -- 172.21.15.59:0/3211551228 shutdown_connections 2024-03-19T03:07:35.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.883+0000 7f905deb0700 1 --2- 172.21.15.59:0/3211551228 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f9040036880 0x7f9040038d30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:35.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.883+0000 7f905deb0700 1 --2- 172.21.15.59:0/3211551228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90580fb950 0x7f905810bfb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:35.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.883+0000 7f905deb0700 1 -- 172.21.15.59:0/3211551228 >> 172.21.15.59:0/3211551228 conn(0x7f90580f7090 msgr2=0x7f9058102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:35.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.883+0000 7f905deb0700 1 -- 172.21.15.59:0/3211551228 shutdown_connections 2024-03-19T03:07:35.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:35.883+0000 7f905deb0700 1 -- 172.21.15.59:0/3211551228 wait complete. 2024-03-19T03:07:36.627 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi176 2024-03-19T03:07:36.627 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-19T03:07:36.627 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/etc/ceph/ceph.conf 2024-03-19T03:07:36.647 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-19T03:07:36.647 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T03:07:36.705 INFO:tasks.cephadm:Adding host smithi176 to orchestrator... 2024-03-19T03:07:36.705 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph orch host add smithi176 2024-03-19T03:07:37.106 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:07:37.229 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:36 smithi059 ceph-mon[29133]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:07:37.230 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:36 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:38.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.648+0000 7ff47867e700 1 -- 172.21.15.59:0/1275404350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff470066580 msgr2=0x7ff4700ff6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:38.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.648+0000 7ff47867e700 1 --2- 172.21.15.59:0/1275404350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff470066580 0x7ff4700ff6b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff46c0048c0 tx=0x7ff46c009e30 comp rx=0 tx=0).stop 2024-03-19T03:07:38.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.648+0000 7ff47867e700 1 -- 172.21.15.59:0/1275404350 shutdown_connections 2024-03-19T03:07:38.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.648+0000 7ff47867e700 1 --2- 172.21.15.59:0/1275404350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff470066580 0x7ff4700ff6b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:38.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.648+0000 7ff47867e700 1 -- 172.21.15.59:0/1275404350 >> 172.21.15.59:0/1275404350 conn(0x7ff4700f70c0 msgr2=0x7ff4700f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:38.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.648+0000 7ff47867e700 1 -- 172.21.15.59:0/1275404350 shutdown_connections 2024-03-19T03:07:38.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.648+0000 7ff47867e700 1 -- 172.21.15.59:0/1275404350 wait complete. 2024-03-19T03:07:38.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.649+0000 7ff47867e700 1 Processor -- start 2024-03-19T03:07:38.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.649+0000 7ff47867e700 1 -- start start 2024-03-19T03:07:38.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.649+0000 7ff47867e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff470066580 0x7ff4700fd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:38.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.649+0000 7ff47867e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4700fd540 con 0x7ff470066580 2024-03-19T03:07:38.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.649+0000 7ff47641a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff470066580 0x7ff4700fd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:38.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.649+0000 7ff47641a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff470066580 0x7ff4700fd000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56036/0 (socket says 172.21.15.59:56036) 2024-03-19T03:07:38.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.649+0000 7ff47641a700 1 -- 172.21.15.59:0/1023527423 learned_addr learned my addr 172.21.15.59:0/1023527423 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:38.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.650+0000 7ff47641a700 1 -- 172.21.15.59:0/1023527423 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff46c005040 con 0x7ff470066580 2024-03-19T03:07:38.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.650+0000 7ff47641a700 1 --2- 172.21.15.59:0/1023527423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff470066580 0x7ff4700fd000 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff46c00b900 tx=0x7ff46c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:38.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.650+0000 7ff466ffd700 1 -- 172.21.15.59:0/1023527423 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff46c00b5c0 con 0x7ff470066580 2024-03-19T03:07:38.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.650+0000 7ff47867e700 1 -- 172.21.15.59:0/1023527423 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4700fb610 con 0x7ff470066580 2024-03-19T03:07:38.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.650+0000 7ff47867e700 1 -- 172.21.15.59:0/1023527423 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4700fbb00 con 0x7ff470066580 2024-03-19T03:07:38.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.650+0000 7ff466ffd700 1 -- 172.21.15.59:0/1023527423 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff46c00be70 con 0x7ff470066580 2024-03-19T03:07:38.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.650+0000 7ff466ffd700 1 -- 172.21.15.59:0/1023527423 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff46c01c590 con 0x7ff470066580 2024-03-19T03:07:38.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.651+0000 7ff466ffd700 1 -- 172.21.15.59:0/1023527423 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7ff46c01c800 con 0x7ff470066580 2024-03-19T03:07:38.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.651+0000 7ff466ffd700 1 --2- 172.21.15.59:0/1023527423 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7ff45c0368d0 0x7ff45c038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:38.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.651+0000 7ff466ffd700 1 -- 172.21.15.59:0/1023527423 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff46c04a380 con 0x7ff470066580 2024-03-19T03:07:38.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.651+0000 7ff475c19700 1 --2- 172.21.15.59:0/1023527423 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7ff45c0368d0 0x7ff45c038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:38.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.651+0000 7ff47867e700 1 -- 172.21.15.59:0/1023527423 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff47005f5f0 con 0x7ff470066580 2024-03-19T03:07:38.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.652+0000 7ff475c19700 1 --2- 172.21.15.59:0/1023527423 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7ff45c0368d0 0x7ff45c038d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff460009990 tx=0x7ff460006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:38.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.656+0000 7ff466ffd700 1 -- 172.21.15.59:0/1023527423 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7ff46c021070 con 0x7ff470066580 2024-03-19T03:07:38.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:38.856+0000 7ff47867e700 1 -- 172.21.15.59:0/1023527423 --> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi176", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4700fbd40 con 0x7ff45c0368d0 2024-03-19T03:07:39.152 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:39 smithi059 ceph-mon[29133]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi176", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:07:39.153 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:39 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:39.153 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:39 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:39.153 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:39 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:39.153 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:39 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:40.388 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:40 smithi059 ceph-mon[29133]: Deploying daemon node-exporter.smithi059 on smithi059 2024-03-19T03:07:40.388 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:40 smithi059 ceph-mon[29133]: Deploying cephadm binary to smithi176 2024-03-19T03:07:41.301 INFO:teuthology.orchestra.run.smithi059.stdout:Added host 'smithi176' with addr '172.21.15.176' 2024-03-19T03:07:41.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:41.297+0000 7ff466ffd700 1 -- 172.21.15.59:0/1023527423 <== mgr.14162 v2:172.21.15.59:6800/1857243897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7ff4700fbd40 con 0x7ff45c0368d0 2024-03-19T03:07:41.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:41.300+0000 7ff47867e700 1 -- 172.21.15.59:0/1023527423 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7ff45c0368d0 msgr2=0x7ff45c038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:41.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:41.300+0000 7ff47867e700 1 --2- 172.21.15.59:0/1023527423 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7ff45c0368d0 0x7ff45c038d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff460009990 tx=0x7ff460006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:41.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:41.300+0000 7ff47867e700 1 -- 172.21.15.59:0/1023527423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff470066580 msgr2=0x7ff4700fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:41.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:41.300+0000 7ff47867e700 1 --2- 172.21.15.59:0/1023527423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff470066580 0x7ff4700fd000 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff46c00b900 tx=0x7ff46c009e30 comp rx=0 tx=0).stop 2024-03-19T03:07:41.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:41.300+0000 7ff47867e700 1 -- 172.21.15.59:0/1023527423 shutdown_connections 2024-03-19T03:07:41.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:41.300+0000 7ff47867e700 1 --2- 172.21.15.59:0/1023527423 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7ff45c0368d0 0x7ff45c038d80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:41.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:41.300+0000 7ff47867e700 1 --2- 172.21.15.59:0/1023527423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff470066580 0x7ff4700fd000 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:41.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:41.300+0000 7ff47867e700 1 -- 172.21.15.59:0/1023527423 >> 172.21.15.59:0/1023527423 conn(0x7ff4700f70c0 msgr2=0x7ff470103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:41.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:41.300+0000 7ff47867e700 1 -- 172.21.15.59:0/1023527423 shutdown_connections 2024-03-19T03:07:41.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:41.301+0000 7ff47867e700 1 -- 172.21.15.59:0/1023527423 wait complete. 2024-03-19T03:07:42.056 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:41 smithi059 ceph-mon[29133]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:07:42.057 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:41 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:42.057 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:41 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:42.362 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-19T03:07:42.812 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:07:42.956 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:42 smithi059 ceph-mon[29133]: Added host smithi176 2024-03-19T03:07:44.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:43 smithi059 ceph-mon[29133]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:07:44.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:43 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:44.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:43 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:44.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:43 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:44.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:43 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:44.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:43 smithi059 ceph-mon[29133]: Deploying daemon alertmanager.smithi059 on smithi059 2024-03-19T03:07:44.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.458+0000 7f50c1991700 1 -- 172.21.15.59:0/3292976886 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50bc0fca50 msgr2=0x7f50bc0fce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:44.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.458+0000 7f50c1991700 1 --2- 172.21.15.59:0/3292976886 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50bc0fca50 0x7f50bc0fce60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0048c0 tx=0x7f50ac009e30 comp rx=0 tx=0).stop 2024-03-19T03:07:44.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.458+0000 7f50c1991700 1 -- 172.21.15.59:0/3292976886 shutdown_connections 2024-03-19T03:07:44.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.458+0000 7f50c1991700 1 --2- 172.21.15.59:0/3292976886 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50bc0fca50 0x7f50bc0fce60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:44.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.458+0000 7f50c1991700 1 -- 172.21.15.59:0/3292976886 >> 172.21.15.59:0/3292976886 conn(0x7f50bc0f8190 msgr2=0x7f50bc0fa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:44.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.458+0000 7f50c1991700 1 -- 172.21.15.59:0/3292976886 shutdown_connections 2024-03-19T03:07:44.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.458+0000 7f50c1991700 1 -- 172.21.15.59:0/3292976886 wait complete. 2024-03-19T03:07:44.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.459+0000 7f50c1991700 1 Processor -- start 2024-03-19T03:07:44.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.459+0000 7f50c1991700 1 -- start start 2024-03-19T03:07:44.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.459+0000 7f50c1991700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50bc0fca50 0x7f50bc10d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:44.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.459+0000 7f50c1991700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc10d5f0 con 0x7f50bc0fca50 2024-03-19T03:07:44.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.459+0000 7f50baffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50bc0fca50 0x7f50bc10d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:44.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.459+0000 7f50baffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50bc0fca50 0x7f50bc10d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50172/0 (socket says 172.21.15.59:50172) 2024-03-19T03:07:44.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.459+0000 7f50baffd700 1 -- 172.21.15.59:0/748081715 learned_addr learned my addr 172.21.15.59:0/748081715 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:44.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.459+0000 7f50baffd700 1 -- 172.21.15.59:0/748081715 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ac005040 con 0x7f50bc0fca50 2024-03-19T03:07:44.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.459+0000 7f50baffd700 1 --2- 172.21.15.59:0/748081715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50bc0fca50 0x7f50bc10d0b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0096b0 tx=0x7f50ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:44.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.460+0000 7f50a37fe700 1 -- 172.21.15.59:0/748081715 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50ac00bed0 con 0x7f50bc0fca50 2024-03-19T03:07:44.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.460+0000 7f50c1991700 1 -- 172.21.15.59:0/748081715 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50bc10d7f0 con 0x7f50bc0fca50 2024-03-19T03:07:44.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.460+0000 7f50a37fe700 1 -- 172.21.15.59:0/748081715 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f50ac00b5c0 con 0x7f50bc0fca50 2024-03-19T03:07:44.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.460+0000 7f50a37fe700 1 -- 172.21.15.59:0/748081715 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50ac01c5b0 con 0x7f50bc0fca50 2024-03-19T03:07:44.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.460+0000 7f50c1991700 1 -- 172.21.15.59:0/748081715 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50bc10e360 con 0x7f50bc0fca50 2024-03-19T03:07:44.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.461+0000 7f50a37fe700 1 -- 172.21.15.59:0/748081715 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f50ac01c820 con 0x7f50bc0fca50 2024-03-19T03:07:44.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.461+0000 7f50a37fe700 1 --2- 172.21.15.59:0/748081715 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f50a4036880 0x7f50a4038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:44.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.461+0000 7f50c1991700 1 -- 172.21.15.59:0/748081715 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50bc102f60 con 0x7f50bc0fca50 2024-03-19T03:07:44.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.461+0000 7f50a37fe700 1 -- 172.21.15.59:0/748081715 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f50ac04aa40 con 0x7f50bc0fca50 2024-03-19T03:07:44.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.461+0000 7f50ba7fc700 1 --2- 172.21.15.59:0/748081715 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f50a4036880 0x7f50a4038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:44.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.461+0000 7f50ba7fc700 1 --2- 172.21.15.59:0/748081715 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f50a4036880 0x7f50a4038d30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009990 tx=0x7f50b0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:44.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.466+0000 7f50a37fe700 1 -- 172.21.15.59:0/748081715 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f50ac023300 con 0x7f50bc0fca50 2024-03-19T03:07:44.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.672+0000 7f50c1991700 1 -- 172.21.15.59:0/748081715 --> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f50bc10e4f0 con 0x7f50a4036880 2024-03-19T03:07:44.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.673+0000 7f50a37fe700 1 -- 172.21.15.59:0/748081715 <== mgr.14162 v2:172.21.15.59:6800/1857243897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f50bc10e4f0 con 0x7f50a4036880 2024-03-19T03:07:44.675 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:07:44.675 INFO:teuthology.orchestra.run.smithi059.stdout:[{"addr": "172.21.15.59", "hostname": "smithi059", "labels": [], "status": ""}, {"addr": "172.21.15.176", "hostname": "smithi176", "labels": [], "status": ""}] 2024-03-19T03:07:44.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.675+0000 7f50c1991700 1 -- 172.21.15.59:0/748081715 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f50a4036880 msgr2=0x7f50a4038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:44.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.675+0000 7f50c1991700 1 --2- 172.21.15.59:0/748081715 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f50a4036880 0x7f50a4038d30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009990 tx=0x7f50b0006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:44.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.675+0000 7f50c1991700 1 -- 172.21.15.59:0/748081715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50bc0fca50 msgr2=0x7f50bc10d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:44.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.675+0000 7f50c1991700 1 --2- 172.21.15.59:0/748081715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50bc0fca50 0x7f50bc10d0b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0096b0 tx=0x7f50ac009e30 comp rx=0 tx=0).stop 2024-03-19T03:07:44.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.675+0000 7f50c1991700 1 -- 172.21.15.59:0/748081715 shutdown_connections 2024-03-19T03:07:44.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.675+0000 7f50c1991700 1 --2- 172.21.15.59:0/748081715 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f50a4036880 0x7f50a4038d30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:44.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.675+0000 7f50c1991700 1 --2- 172.21.15.59:0/748081715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50bc0fca50 0x7f50bc10d0b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:44.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.675+0000 7f50c1991700 1 -- 172.21.15.59:0/748081715 >> 172.21.15.59:0/748081715 conn(0x7f50bc0f8190 msgr2=0x7f50bc103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:44.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.675+0000 7f50c1991700 1 -- 172.21.15.59:0/748081715 shutdown_connections 2024-03-19T03:07:44.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:44.675+0000 7f50c1991700 1 -- 172.21.15.59:0/748081715 wait complete. 2024-03-19T03:07:45.454 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-19T03:07:45.454 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-19T03:07:45.710 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:07:46.081 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:45 smithi059 ceph-mon[29133]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T03:07:46.082 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:45 smithi059 ceph-mon[29133]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:07:46.082 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:45 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:48.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:47 smithi059 ceph-mon[29133]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:07:48.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.265+0000 7fc2aa518700 1 -- 172.21.15.59:0/2099468551 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2a40fd050 msgr2=0x7fc2a40fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:48.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.265+0000 7fc2aa518700 1 --2- 172.21.15.59:0/2099468551 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2a40fd050 0x7fc2a40fd460 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc2940048c0 tx=0x7fc294009e30 comp rx=0 tx=0).stop 2024-03-19T03:07:48.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.265+0000 7fc2aa518700 1 -- 172.21.15.59:0/2099468551 shutdown_connections 2024-03-19T03:07:48.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.265+0000 7fc2aa518700 1 --2- 172.21.15.59:0/2099468551 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2a40fd050 0x7fc2a40fd460 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.265+0000 7fc2aa518700 1 -- 172.21.15.59:0/2099468551 >> 172.21.15.59:0/2099468551 conn(0x7fc2a40f70d0 msgr2=0x7fc2a40f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.265+0000 7fc2aa518700 1 -- 172.21.15.59:0/2099468551 shutdown_connections 2024-03-19T03:07:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.265+0000 7fc2aa518700 1 -- 172.21.15.59:0/2099468551 wait complete. 2024-03-19T03:07:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.265+0000 7fc2aa518700 1 Processor -- start 2024-03-19T03:07:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.265+0000 7fc2aa518700 1 -- start start 2024-03-19T03:07:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.266+0000 7fc2aa518700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2a40fd050 0x7fc2a4065bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.266+0000 7fc2aa518700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a410fde0 con 0x7fc2a40fd050 2024-03-19T03:07:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.266+0000 7fc2a3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2a40fd050 0x7fc2a4065bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.266+0000 7fc2a3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2a40fd050 0x7fc2a4065bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50198/0 (socket says 172.21.15.59:50198) 2024-03-19T03:07:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.266+0000 7fc2a3fff700 1 -- 172.21.15.59:0/822620716 learned_addr learned my addr 172.21.15.59:0/822620716 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:07:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.266+0000 7fc2a3fff700 1 -- 172.21.15.59:0/822620716 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc294005040 con 0x7fc2a40fd050 2024-03-19T03:07:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.266+0000 7fc2a3fff700 1 --2- 172.21.15.59:0/822620716 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2a40fd050 0x7fc2a4065bd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc2940096b0 tx=0x7fc294009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.266+0000 7fc2a0ff9700 1 -- 172.21.15.59:0/822620716 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc29400bed0 con 0x7fc2a40fd050 2024-03-19T03:07:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.266+0000 7fc2a0ff9700 1 -- 172.21.15.59:0/822620716 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc29400b5c0 con 0x7fc2a40fd050 2024-03-19T03:07:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.266+0000 7fc2aa518700 1 -- 172.21.15.59:0/822620716 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a410ff20 con 0x7fc2a40fd050 2024-03-19T03:07:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.266+0000 7fc2a0ff9700 1 -- 172.21.15.59:0/822620716 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc29401c5b0 con 0x7fc2a40fd050 2024-03-19T03:07:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.266+0000 7fc2aa518700 1 -- 172.21.15.59:0/822620716 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a4110410 con 0x7fc2a40fd050 2024-03-19T03:07:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.267+0000 7fc2a0ff9700 1 -- 172.21.15.59:0/822620716 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fc294013420 con 0x7fc2a40fd050 2024-03-19T03:07:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.267+0000 7fc2aa518700 1 -- 172.21.15.59:0/822620716 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a4105920 con 0x7fc2a40fd050 2024-03-19T03:07:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.267+0000 7fc2a0ff9700 1 --2- 172.21.15.59:0/822620716 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fc28c036880 0x7fc28c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.267+0000 7fc2a0ff9700 1 -- 172.21.15.59:0/822620716 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc294049ad0 con 0x7fc2a40fd050 2024-03-19T03:07:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.268+0000 7fc2a37fe700 1 --2- 172.21.15.59:0/822620716 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fc28c036880 0x7fc28c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.268+0000 7fc2a37fe700 1 --2- 172.21.15.59:0/822620716 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fc28c036880 0x7fc28c038d30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc298009990 tx=0x7fc298006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:48.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.273+0000 7fc2a0ff9700 1 -- 172.21.15.59:0/822620716 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fc294013a80 con 0x7fc2a40fd050 2024-03-19T03:07:48.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.483+0000 7fc2aa518700 1 -- 172.21.15.59:0/822620716 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc2a405f5f0 con 0x7fc2a40fd050 2024-03-19T03:07:48.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.826+0000 7fc2a0ff9700 1 -- 172.21.15.59:0/822620716 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc294011370 con 0x7fc2a40fd050 2024-03-19T03:07:48.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.829+0000 7fc2aa518700 1 -- 172.21.15.59:0/822620716 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fc28c036880 msgr2=0x7fc28c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:48.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.829+0000 7fc2aa518700 1 --2- 172.21.15.59:0/822620716 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fc28c036880 0x7fc28c038d30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc298009990 tx=0x7fc298006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:48.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.829+0000 7fc2aa518700 1 -- 172.21.15.59:0/822620716 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2a40fd050 msgr2=0x7fc2a4065bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:48.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.829+0000 7fc2aa518700 1 --2- 172.21.15.59:0/822620716 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2a40fd050 0x7fc2a4065bd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc2940096b0 tx=0x7fc294009e30 comp rx=0 tx=0).stop 2024-03-19T03:07:48.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.829+0000 7fc2aa518700 1 -- 172.21.15.59:0/822620716 shutdown_connections 2024-03-19T03:07:48.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.829+0000 7fc2aa518700 1 --2- 172.21.15.59:0/822620716 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fc28c036880 0x7fc28c038d30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:48.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.829+0000 7fc2aa518700 1 --2- 172.21.15.59:0/822620716 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2a40fd050 0x7fc2a4065bd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:48.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.829+0000 7fc2aa518700 1 -- 172.21.15.59:0/822620716 >> 172.21.15.59:0/822620716 conn(0x7fc2a40f70d0 msgr2=0x7fc2a4102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:48.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.829+0000 7fc2aa518700 1 -- 172.21.15.59:0/822620716 shutdown_connections 2024-03-19T03:07:48.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:07:48.829+0000 7fc2aa518700 1 -- 172.21.15.59:0/822620716 wait complete. 2024-03-19T03:07:48.832 INFO:teuthology.orchestra.run.smithi059.stderr:adjusted tunables profile to default 2024-03-19T03:07:49.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:48 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/822620716' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-19T03:07:49.696 INFO:tasks.cephadm:Adding mon.smithi059 on smithi059 2024-03-19T03:07:49.696 INFO:tasks.cephadm:Adding mon.smithi176 on smithi176 2024-03-19T03:07:49.696 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph orch apply mon '2;smithi059:172.21.15.59=smithi059;smithi176:172.21.15.176=smithi176' 2024-03-19T03:07:49.946 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:07:49.992 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:07:50.042 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:49 smithi059 ceph-mon[29133]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:07:50.042 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:49 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/822620716' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-19T03:07:50.042 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:49 smithi059 ceph-mon[29133]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T03:07:51.226 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:51 smithi059 ceph-mon[29133]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:07:51.454 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.450+0000 7fcd78014700 1 -- 172.21.15.176:0/991922675 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd700fca60 msgr2=0x7fcd700fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:51.454 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.450+0000 7fcd78014700 1 --2- 172.21.15.176:0/991922675 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd700fca60 0x7fcd700fce70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c0048c0 tx=0x7fcd6c009e30 comp rx=0 tx=0).stop 2024-03-19T03:07:51.454 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.450+0000 7fcd78014700 1 -- 172.21.15.176:0/991922675 shutdown_connections 2024-03-19T03:07:51.454 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.450+0000 7fcd78014700 1 --2- 172.21.15.176:0/991922675 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd700fca60 0x7fcd700fce70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:51.454 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.450+0000 7fcd78014700 1 -- 172.21.15.176:0/991922675 >> 172.21.15.176:0/991922675 conn(0x7fcd700f81c0 msgr2=0x7fcd700fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:51.454 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.450+0000 7fcd78014700 1 -- 172.21.15.176:0/991922675 shutdown_connections 2024-03-19T03:07:51.454 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.450+0000 7fcd78014700 1 -- 172.21.15.176:0/991922675 wait complete. 2024-03-19T03:07:51.454 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.451+0000 7fcd78014700 1 Processor -- start 2024-03-19T03:07:51.454 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.451+0000 7fcd78014700 1 -- start start 2024-03-19T03:07:51.455 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.451+0000 7fcd78014700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd700fca60 0x7fcd70071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:51.455 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.451+0000 7fcd78014700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd700722a0 con 0x7fcd700fca60 2024-03-19T03:07:51.455 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.451+0000 7fcd75db0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd700fca60 0x7fcd70071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:51.455 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.451+0000 7fcd75db0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd700fca60 0x7fcd70071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.176:50504/0 (socket says 172.21.15.176:50504) 2024-03-19T03:07:51.455 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.451+0000 7fcd75db0700 1 -- 172.21.15.176:0/3202615037 learned_addr learned my addr 172.21.15.176:0/3202615037 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:07:51.455 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.451+0000 7fcd75db0700 1 -- 172.21.15.176:0/3202615037 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd6c005040 con 0x7fcd700fca60 2024-03-19T03:07:51.455 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.451+0000 7fcd75db0700 1 --2- 172.21.15.176:0/3202615037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd700fca60 0x7fcd70071d60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c0096b0 tx=0x7fcd6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:51.455 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.452+0000 7fcd667fc700 1 -- 172.21.15.176:0/3202615037 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd6c00bed0 con 0x7fcd700fca60 2024-03-19T03:07:51.455 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.452+0000 7fcd78014700 1 -- 172.21.15.176:0/3202615037 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd70070370 con 0x7fcd700fca60 2024-03-19T03:07:51.455 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.452+0000 7fcd667fc700 1 -- 172.21.15.176:0/3202615037 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd6c00b5c0 con 0x7fcd700fca60 2024-03-19T03:07:51.455 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.452+0000 7fcd78014700 1 -- 172.21.15.176:0/3202615037 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd70070860 con 0x7fcd700fca60 2024-03-19T03:07:51.455 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.452+0000 7fcd667fc700 1 -- 172.21.15.176:0/3202615037 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd6c025560 con 0x7fcd700fca60 2024-03-19T03:07:51.456 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.453+0000 7fcd667fc700 1 -- 172.21.15.176:0/3202615037 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fcd6c013a80 con 0x7fcd700fca60 2024-03-19T03:07:51.456 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.453+0000 7fcd78014700 1 -- 172.21.15.176:0/3202615037 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd7005f5f0 con 0x7fcd700fca60 2024-03-19T03:07:51.456 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.453+0000 7fcd667fc700 1 --2- 172.21.15.176:0/3202615037 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fcd5c036880 0x7fcd5c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:51.456 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.453+0000 7fcd667fc700 1 -- 172.21.15.176:0/3202615037 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fcd6c049aa0 con 0x7fcd700fca60 2024-03-19T03:07:51.456 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.453+0000 7fcd755af700 1 --2- 172.21.15.176:0/3202615037 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fcd5c036880 0x7fcd5c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:51.456 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.453+0000 7fcd755af700 1 --2- 172.21.15.176:0/3202615037 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fcd5c036880 0x7fcd5c038d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fcd60009990 tx=0x7fcd60006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:51.460 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.458+0000 7fcd667fc700 1 -- 172.21.15.176:0/3202615037 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fcd6c011550 con 0x7fcd700fca60 2024-03-19T03:07:51.650 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.648+0000 7fcd78014700 1 -- 172.21.15.176:0/3202615037 --> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi059:172.21.15.59=smithi059;smithi176:172.21.15.176=smithi176", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd70106560 con 0x7fcd5c036880 2024-03-19T03:07:51.657 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.656+0000 7fcd667fc700 1 -- 172.21.15.176:0/3202615037 <== mgr.14162 v2:172.21.15.59:6800/1857243897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fcd70106560 con 0x7fcd5c036880 2024-03-19T03:07:51.657 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled mon update... 2024-03-19T03:07:51.659 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.657+0000 7fcd78014700 1 -- 172.21.15.176:0/3202615037 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fcd5c036880 msgr2=0x7fcd5c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:51.659 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.657+0000 7fcd78014700 1 --2- 172.21.15.176:0/3202615037 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fcd5c036880 0x7fcd5c038d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fcd60009990 tx=0x7fcd60006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:51.659 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.657+0000 7fcd78014700 1 -- 172.21.15.176:0/3202615037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd700fca60 msgr2=0x7fcd70071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:51.659 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.657+0000 7fcd78014700 1 --2- 172.21.15.176:0/3202615037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd700fca60 0x7fcd70071d60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c0096b0 tx=0x7fcd6c009e30 comp rx=0 tx=0).stop 2024-03-19T03:07:51.659 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.657+0000 7fcd78014700 1 -- 172.21.15.176:0/3202615037 shutdown_connections 2024-03-19T03:07:51.660 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.657+0000 7fcd78014700 1 --2- 172.21.15.176:0/3202615037 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fcd5c036880 0x7fcd5c038d30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:51.660 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.657+0000 7fcd78014700 1 --2- 172.21.15.176:0/3202615037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd700fca60 0x7fcd70071d60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:51.660 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.657+0000 7fcd78014700 1 -- 172.21.15.176:0/3202615037 >> 172.21.15.176:0/3202615037 conn(0x7fcd700f81c0 msgr2=0x7fcd70103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:51.660 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.657+0000 7fcd78014700 1 -- 172.21.15.176:0/3202615037 shutdown_connections 2024-03-19T03:07:51.660 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:51.657+0000 7fcd78014700 1 -- 172.21.15.176:0/3202615037 wait complete. 2024-03-19T03:07:52.270 DEBUG:teuthology.orchestra.run.smithi176:mon.smithi176> sudo journalctl -f -n 0 -u ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@mon.smithi176.service 2024-03-19T03:07:52.273 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-19T03:07:52.273 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T03:07:52.673 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:07:52.722 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:07:52.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:52 smithi059 ceph-mon[29133]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi059:172.21.15.59=smithi059;smithi176:172.21.15.176=smithi176", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:07:52.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:52 smithi059 ceph-mon[29133]: Saving service mon spec with placement smithi059:172.21.15.59=smithi059;smithi176:172.21.15.176=smithi176;count:2 2024-03-19T03:07:52.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:52 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:53.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.975+0000 7f7972502700 1 -- 172.21.15.176:0/2756588842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f796c066960 msgr2=0x7f796c0ff6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:53.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.975+0000 7f7972502700 1 --2- 172.21.15.176:0/2756588842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f796c066960 0x7f796c0ff6a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f795c0048c0 tx=0x7f795c009e30 comp rx=0 tx=0).stop 2024-03-19T03:07:53.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.976+0000 7f7972502700 1 -- 172.21.15.176:0/2756588842 shutdown_connections 2024-03-19T03:07:53.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.976+0000 7f7972502700 1 --2- 172.21.15.176:0/2756588842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f796c066960 0x7f796c0ff6a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:53.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.976+0000 7f7972502700 1 -- 172.21.15.176:0/2756588842 >> 172.21.15.176:0/2756588842 conn(0x7f796c0f7090 msgr2=0x7f796c0f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:53.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.976+0000 7f7972502700 1 -- 172.21.15.176:0/2756588842 shutdown_connections 2024-03-19T03:07:53.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.976+0000 7f7972502700 1 -- 172.21.15.176:0/2756588842 wait complete. 2024-03-19T03:07:53.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.976+0000 7f7972502700 1 Processor -- start 2024-03-19T03:07:53.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.976+0000 7f7972502700 1 -- start start 2024-03-19T03:07:53.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.976+0000 7f7972502700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f796c066960 0x7f796c0fcff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:53.980 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.976+0000 7f7972502700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c0fd530 con 0x7f796c066960 2024-03-19T03:07:53.980 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.977+0000 7f796bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f796c066960 0x7f796c0fcff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:53.980 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.977+0000 7f796bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f796c066960 0x7f796c0fcff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.176:50518/0 (socket says 172.21.15.176:50518) 2024-03-19T03:07:53.980 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.977+0000 7f796bfff700 1 -- 172.21.15.176:0/1732987097 learned_addr learned my addr 172.21.15.176:0/1732987097 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:07:53.981 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.977+0000 7f796bfff700 1 -- 172.21.15.176:0/1732987097 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f795c005040 con 0x7f796c066960 2024-03-19T03:07:53.981 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.977+0000 7f796bfff700 1 --2- 172.21.15.176:0/1732987097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f796c066960 0x7f796c0fcff0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f795c009680 tx=0x7f795c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:53.981 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.977+0000 7f7968ff9700 1 -- 172.21.15.176:0/1732987097 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f795c00bed0 con 0x7f796c066960 2024-03-19T03:07:53.981 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.977+0000 7f7972502700 1 -- 172.21.15.176:0/1732987097 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f796c0fb600 con 0x7f796c066960 2024-03-19T03:07:53.981 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.977+0000 7f7968ff9700 1 -- 172.21.15.176:0/1732987097 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f795c00b5c0 con 0x7f796c066960 2024-03-19T03:07:53.982 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.977+0000 7f7968ff9700 1 -- 172.21.15.176:0/1732987097 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f795c01c5b0 con 0x7f796c066960 2024-03-19T03:07:53.982 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.977+0000 7f7972502700 1 -- 172.21.15.176:0/1732987097 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f796c0fbaf0 con 0x7f796c066960 2024-03-19T03:07:53.982 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.978+0000 7f7968ff9700 1 -- 172.21.15.176:0/1732987097 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f795c013420 con 0x7f796c066960 2024-03-19T03:07:53.982 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.978+0000 7f7972502700 1 -- 172.21.15.176:0/1732987097 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f796c05f5f0 con 0x7f796c066960 2024-03-19T03:07:53.982 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.978+0000 7f7968ff9700 1 --2- 172.21.15.176:0/1732987097 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f79540368d0 0x7f7954038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:53.982 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.978+0000 7f7968ff9700 1 -- 172.21.15.176:0/1732987097 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f795c0499f0 con 0x7f796c066960 2024-03-19T03:07:53.982 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.978+0000 7f796b7fe700 1 --2- 172.21.15.176:0/1732987097 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f79540368d0 0x7f7954038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:53.982 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.979+0000 7f796b7fe700 1 --2- 172.21.15.176:0/1732987097 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f79540368d0 0x7f7954038d80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7960009990 tx=0x7f7960006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:53.985 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:53.983+0000 7f7968ff9700 1 -- 172.21.15.176:0/1732987097 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f795c011550 con 0x7f796c066960 2024-03-19T03:07:54.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:53 smithi059 ceph-mon[29133]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:07:54.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:53 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:54.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:53 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:54.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:53 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:54.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:53 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:54.240 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:54.238+0000 7f7972502700 1 -- 172.21.15.176:0/1732987097 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f796c106420 con 0x7f796c066960 2024-03-19T03:07:54.241 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:54.239+0000 7f7968ff9700 1 -- 172.21.15.176:0/1732987097 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f795c011370 con 0x7f796c066960 2024-03-19T03:07:54.241 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T03:07:54.241 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":1,"fsid":"8c8ae530-e59d-11ee-95c9-87774f69a715","modified":"2024-03-19T03:06:12.321211Z","created":"2024-03-19T03:06:12.321211Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-19T03:07:54.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:54.241+0000 7f7972502700 1 -- 172.21.15.176:0/1732987097 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f79540368d0 msgr2=0x7f7954038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:54.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:54.241+0000 7f7972502700 1 --2- 172.21.15.176:0/1732987097 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f79540368d0 0x7f7954038d80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7960009990 tx=0x7f7960006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:54.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:54.241+0000 7f7972502700 1 -- 172.21.15.176:0/1732987097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f796c066960 msgr2=0x7f796c0fcff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:54.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:54.241+0000 7f7972502700 1 --2- 172.21.15.176:0/1732987097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f796c066960 0x7f796c0fcff0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f795c009680 tx=0x7f795c009e30 comp rx=0 tx=0).stop 2024-03-19T03:07:54.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:54.241+0000 7f7972502700 1 -- 172.21.15.176:0/1732987097 shutdown_connections 2024-03-19T03:07:54.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:54.241+0000 7f7972502700 1 --2- 172.21.15.176:0/1732987097 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f79540368d0 0x7f7954038d80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:54.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:54.241+0000 7f7972502700 1 --2- 172.21.15.176:0/1732987097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f796c066960 0x7f796c0fcff0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:54.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:54.241+0000 7f7972502700 1 -- 172.21.15.176:0/1732987097 >> 172.21.15.176:0/1732987097 conn(0x7f796c0f7090 msgr2=0x7f796c103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:54.244 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:54.241+0000 7f7972502700 1 -- 172.21.15.176:0/1732987097 shutdown_connections 2024-03-19T03:07:54.244 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:54.241+0000 7f7972502700 1 -- 172.21.15.176:0/1732987097 wait complete. 2024-03-19T03:07:54.244 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 1 2024-03-19T03:07:55.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:54 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:55.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:54 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:55.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:54 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-19T03:07:55.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:54 smithi059 ceph-mon[29133]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-19T03:07:55.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:54 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:55.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:54 smithi059 ceph-mon[29133]: Deploying daemon grafana.smithi059 on smithi059 2024-03-19T03:07:55.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:54 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/1732987097' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:07:55.784 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-19T03:07:55.785 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T03:07:56.035 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:07:56.081 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:07:56.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:55 smithi059 ceph-mon[29133]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:07:57.001 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:56 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:07:57.502 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.498+0000 7f36297a1700 1 -- 172.21.15.176:0/2344466880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36240fb950 msgr2=0x7f36240fbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:57.502 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.498+0000 7f36297a1700 1 --2- 172.21.15.176:0/2344466880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36240fb950 0x7f36240fbd60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f36180048c0 tx=0x7f3618009e30 comp rx=0 tx=0).stop 2024-03-19T03:07:57.502 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.498+0000 7f36297a1700 1 -- 172.21.15.176:0/2344466880 shutdown_connections 2024-03-19T03:07:57.502 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.498+0000 7f36297a1700 1 --2- 172.21.15.176:0/2344466880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36240fb950 0x7f36240fbd60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:57.502 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.498+0000 7f36297a1700 1 -- 172.21.15.176:0/2344466880 >> 172.21.15.176:0/2344466880 conn(0x7f36240f7090 msgr2=0x7f36240f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:57.502 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.498+0000 7f36297a1700 1 -- 172.21.15.176:0/2344466880 shutdown_connections 2024-03-19T03:07:57.502 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.498+0000 7f36297a1700 1 -- 172.21.15.176:0/2344466880 wait complete. 2024-03-19T03:07:57.502 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.499+0000 7f36297a1700 1 Processor -- start 2024-03-19T03:07:57.502 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.499+0000 7f36297a1700 1 -- start start 2024-03-19T03:07:57.502 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.499+0000 7f36297a1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36240fb950 0x7f36240f9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:57.503 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.499+0000 7f36297a1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36240f9580 con 0x7f36240fb950 2024-03-19T03:07:57.503 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.499+0000 7f3622ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36240fb950 0x7f36240f9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:57.503 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.499+0000 7f3622ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36240fb950 0x7f36240f9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.176:50528/0 (socket says 172.21.15.176:50528) 2024-03-19T03:07:57.503 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.499+0000 7f3622ffd700 1 -- 172.21.15.176:0/1989269426 learned_addr learned my addr 172.21.15.176:0/1989269426 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:07:57.503 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.500+0000 7f3622ffd700 1 -- 172.21.15.176:0/1989269426 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3618005040 con 0x7f36240fb950 2024-03-19T03:07:57.504 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.500+0000 7f3622ffd700 1 --2- 172.21.15.176:0/1989269426 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36240fb950 0x7f36240f9040 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f36180096b0 tx=0x7f3618009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:57.504 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.500+0000 7f360b7fe700 1 -- 172.21.15.176:0/1989269426 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f361800bed0 con 0x7f36240fb950 2024-03-19T03:07:57.504 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.500+0000 7f360b7fe700 1 -- 172.21.15.176:0/1989269426 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f361800b5c0 con 0x7f36240fb950 2024-03-19T03:07:57.504 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.500+0000 7f36297a1700 1 -- 172.21.15.176:0/1989269426 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36240f7650 con 0x7f36240fb950 2024-03-19T03:07:57.504 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.500+0000 7f360b7fe700 1 -- 172.21.15.176:0/1989269426 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f361801c5b0 con 0x7f36240fb950 2024-03-19T03:07:57.505 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.500+0000 7f36297a1700 1 -- 172.21.15.176:0/1989269426 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36240f7b40 con 0x7f36240fb950 2024-03-19T03:07:57.505 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.501+0000 7f360b7fe700 1 -- 172.21.15.176:0/1989269426 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f361801c820 con 0x7f36240fb950 2024-03-19T03:07:57.505 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.501+0000 7f360b7fe700 1 --2- 172.21.15.176:0/1989269426 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f360c036880 0x7f360c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:07:57.505 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.501+0000 7f36297a1700 1 -- 172.21.15.176:0/1989269426 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f362405f5f0 con 0x7f36240fb950 2024-03-19T03:07:57.505 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.501+0000 7f360b7fe700 1 -- 172.21.15.176:0/1989269426 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3618049ab0 con 0x7f36240fb950 2024-03-19T03:07:57.505 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.501+0000 7f36227fc700 1 --2- 172.21.15.176:0/1989269426 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f360c036880 0x7f360c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:07:57.505 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.502+0000 7f36227fc700 1 --2- 172.21.15.176:0/1989269426 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f360c036880 0x7f360c038d30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3614009990 tx=0x7f3614006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:07:57.508 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.506+0000 7f360b7fe700 1 -- 172.21.15.176:0/1989269426 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f3618023300 con 0x7f36240fb950 2024-03-19T03:07:57.766 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.763+0000 7f36297a1700 1 -- 172.21.15.176:0/1989269426 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f362404bed0 con 0x7f36240fb950 2024-03-19T03:07:57.766 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.763+0000 7f360b7fe700 1 -- 172.21.15.176:0/1989269426 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f3618011b10 con 0x7f36240fb950 2024-03-19T03:07:57.766 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T03:07:57.766 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":1,"fsid":"8c8ae530-e59d-11ee-95c9-87774f69a715","modified":"2024-03-19T03:06:12.321211Z","created":"2024-03-19T03:06:12.321211Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-19T03:07:57.768 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.765+0000 7f36297a1700 1 -- 172.21.15.176:0/1989269426 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f360c036880 msgr2=0x7f360c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:57.768 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.765+0000 7f36297a1700 1 --2- 172.21.15.176:0/1989269426 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f360c036880 0x7f360c038d30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3614009990 tx=0x7f3614006e40 comp rx=0 tx=0).stop 2024-03-19T03:07:57.768 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.765+0000 7f36297a1700 1 -- 172.21.15.176:0/1989269426 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36240fb950 msgr2=0x7f36240f9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:07:57.768 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.765+0000 7f36297a1700 1 --2- 172.21.15.176:0/1989269426 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36240fb950 0x7f36240f9040 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f36180096b0 tx=0x7f3618009e30 comp rx=0 tx=0).stop 2024-03-19T03:07:57.768 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.765+0000 7f36297a1700 1 -- 172.21.15.176:0/1989269426 shutdown_connections 2024-03-19T03:07:57.768 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.765+0000 7f36297a1700 1 --2- 172.21.15.176:0/1989269426 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f360c036880 0x7f360c038d30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:57.768 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.765+0000 7f36297a1700 1 --2- 172.21.15.176:0/1989269426 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36240fb950 0x7f36240f9040 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:07:57.768 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.765+0000 7f36297a1700 1 -- 172.21.15.176:0/1989269426 >> 172.21.15.176:0/1989269426 conn(0x7f36240f7090 msgr2=0x7f3624065c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:07:57.768 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.765+0000 7f36297a1700 1 -- 172.21.15.176:0/1989269426 shutdown_connections 2024-03-19T03:07:57.768 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:07:57.765+0000 7f36297a1700 1 -- 172.21.15.176:0/1989269426 wait complete. 2024-03-19T03:07:57.769 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 1 2024-03-19T03:07:58.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:57 smithi059 ceph-mon[29133]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:07:58.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:57 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/1989269426' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:07:59.332 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-19T03:07:59.332 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T03:07:59.582 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:07:59.629 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:08:00.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:07:59 smithi059 ceph-mon[29133]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:01.100 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.097+0000 7f05ce5e0700 1 -- 172.21.15.176:0/1537651157 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05c80fb960 msgr2=0x7f05c80fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:01.101 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.097+0000 7f05ce5e0700 1 --2- 172.21.15.176:0/1537651157 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05c80fb960 0x7f05c80fbd70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f05b80048c0 tx=0x7f05b8009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:01.101 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.097+0000 7f05ce5e0700 1 -- 172.21.15.176:0/1537651157 shutdown_connections 2024-03-19T03:08:01.101 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.097+0000 7f05ce5e0700 1 --2- 172.21.15.176:0/1537651157 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05c80fb960 0x7f05c80fbd70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:01.101 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.097+0000 7f05ce5e0700 1 -- 172.21.15.176:0/1537651157 >> 172.21.15.176:0/1537651157 conn(0x7f05c80f70c0 msgr2=0x7f05c80f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:01.101 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.097+0000 7f05ce5e0700 1 -- 172.21.15.176:0/1537651157 shutdown_connections 2024-03-19T03:08:01.101 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.097+0000 7f05ce5e0700 1 -- 172.21.15.176:0/1537651157 wait complete. 2024-03-19T03:08:01.101 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.097+0000 7f05ce5e0700 1 Processor -- start 2024-03-19T03:08:01.101 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.097+0000 7f05ce5e0700 1 -- start start 2024-03-19T03:08:01.101 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.097+0000 7f05ce5e0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05c80fb960 0x7f05c80f9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:01.101 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.097+0000 7f05ce5e0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c80f9590 con 0x7f05c80fb960 2024-03-19T03:08:01.101 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.098+0000 7f05c7fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05c80fb960 0x7f05c80f9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:01.102 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.098+0000 7f05c7fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05c80fb960 0x7f05c80f9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.176:60678/0 (socket says 172.21.15.176:60678) 2024-03-19T03:08:01.102 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.098+0000 7f05c7fff700 1 -- 172.21.15.176:0/280298192 learned_addr learned my addr 172.21.15.176:0/280298192 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:08:01.102 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.098+0000 7f05c7fff700 1 -- 172.21.15.176:0/280298192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b8005040 con 0x7f05c80fb960 2024-03-19T03:08:01.102 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.098+0000 7f05c7fff700 1 --2- 172.21.15.176:0/280298192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05c80fb960 0x7f05c80f9050 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f05b8009680 tx=0x7f05b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:01.102 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.098+0000 7f05c4ff9700 1 -- 172.21.15.176:0/280298192 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05b800b5c0 con 0x7f05c80fb960 2024-03-19T03:08:01.102 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.098+0000 7f05c4ff9700 1 -- 172.21.15.176:0/280298192 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f05b800be70 con 0x7f05c80fb960 2024-03-19T03:08:01.102 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.098+0000 7f05ce5e0700 1 -- 172.21.15.176:0/280298192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c80f7660 con 0x7f05c80fb960 2024-03-19T03:08:01.102 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.098+0000 7f05c4ff9700 1 -- 172.21.15.176:0/280298192 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05b80135a0 con 0x7f05c80fb960 2024-03-19T03:08:01.102 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.098+0000 7f05ce5e0700 1 -- 172.21.15.176:0/280298192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c80f7b50 con 0x7f05c80fb960 2024-03-19T03:08:01.103 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.099+0000 7f05c4ff9700 1 -- 172.21.15.176:0/280298192 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f05b801c430 con 0x7f05c80fb960 2024-03-19T03:08:01.103 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.099+0000 7f05ce5e0700 1 -- 172.21.15.176:0/280298192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c805f5f0 con 0x7f05c80fb960 2024-03-19T03:08:01.103 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.099+0000 7f05c4ff9700 1 --2- 172.21.15.176:0/280298192 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f05b0036880 0x7f05b0038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:01.103 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.099+0000 7f05c4ff9700 1 -- 172.21.15.176:0/280298192 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f05b8049420 con 0x7f05c80fb960 2024-03-19T03:08:01.103 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.099+0000 7f05c77fe700 1 --2- 172.21.15.176:0/280298192 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f05b0036880 0x7f05b0038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:01.104 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.100+0000 7f05c77fe700 1 --2- 172.21.15.176:0/280298192 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f05b0036880 0x7f05b0038d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f05bc009990 tx=0x7f05bc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:01.107 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.105+0000 7f05c4ff9700 1 -- 172.21.15.176:0/280298192 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f05b8004d30 con 0x7f05c80fb960 2024-03-19T03:08:01.364 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.361+0000 7f05ce5e0700 1 -- 172.21.15.176:0/280298192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f05c804bed0 con 0x7f05c80fb960 2024-03-19T03:08:01.364 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.362+0000 7f05c4ff9700 1 -- 172.21.15.176:0/280298192 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f05b8021070 con 0x7f05c80fb960 2024-03-19T03:08:01.364 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T03:08:01.364 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":1,"fsid":"8c8ae530-e59d-11ee-95c9-87774f69a715","modified":"2024-03-19T03:06:12.321211Z","created":"2024-03-19T03:06:12.321211Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-19T03:08:01.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.364+0000 7f05ce5e0700 1 -- 172.21.15.176:0/280298192 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f05b0036880 msgr2=0x7f05b0038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:01.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.364+0000 7f05ce5e0700 1 --2- 172.21.15.176:0/280298192 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f05b0036880 0x7f05b0038d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f05bc009990 tx=0x7f05bc006e40 comp rx=0 tx=0).stop 2024-03-19T03:08:01.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.364+0000 7f05ce5e0700 1 -- 172.21.15.176:0/280298192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05c80fb960 msgr2=0x7f05c80f9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:01.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.364+0000 7f05ce5e0700 1 --2- 172.21.15.176:0/280298192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05c80fb960 0x7f05c80f9050 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f05b8009680 tx=0x7f05b8009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:01.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.364+0000 7f05ce5e0700 1 -- 172.21.15.176:0/280298192 shutdown_connections 2024-03-19T03:08:01.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.364+0000 7f05ce5e0700 1 --2- 172.21.15.176:0/280298192 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f05b0036880 0x7f05b0038d30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:01.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.364+0000 7f05ce5e0700 1 --2- 172.21.15.176:0/280298192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05c80fb960 0x7f05c80f9050 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:01.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.364+0000 7f05ce5e0700 1 -- 172.21.15.176:0/280298192 >> 172.21.15.176:0/280298192 conn(0x7f05c80f70c0 msgr2=0x7f05c8065ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:01.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.364+0000 7f05ce5e0700 1 -- 172.21.15.176:0/280298192 shutdown_connections 2024-03-19T03:08:01.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:01.364+0000 7f05ce5e0700 1 -- 172.21.15.176:0/280298192 wait complete. 2024-03-19T03:08:01.367 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 1 2024-03-19T03:08:01.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:01 smithi059 ceph-mon[29133]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:02.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:02 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/280298192' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:02.985 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-19T03:08:02.985 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T03:08:03.227 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:08:03.274 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:08:03.392 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:03 smithi059 ceph-mon[29133]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:04.775 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.771+0000 7f719c0f8700 1 -- 172.21.15.176:0/2539970414 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7194071e40 msgr2=0x7f7194070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:04.775 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.771+0000 7f719c0f8700 1 --2- 172.21.15.176:0/2539970414 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7194071e40 0x7f7194070350 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f718c009c60 tx=0x7f718c004d10 comp rx=0 tx=0).stop 2024-03-19T03:08:04.775 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.771+0000 7f719c0f8700 1 -- 172.21.15.176:0/2539970414 shutdown_connections 2024-03-19T03:08:04.775 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.771+0000 7f719c0f8700 1 --2- 172.21.15.176:0/2539970414 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7194071e40 0x7f7194070350 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:04.775 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.771+0000 7f719c0f8700 1 -- 172.21.15.176:0/2539970414 >> 172.21.15.176:0/2539970414 conn(0x7f71940f8190 msgr2=0x7f71940fa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:04.775 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.771+0000 7f719c0f8700 1 -- 172.21.15.176:0/2539970414 shutdown_connections 2024-03-19T03:08:04.775 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.771+0000 7f719c0f8700 1 -- 172.21.15.176:0/2539970414 wait complete. 2024-03-19T03:08:04.775 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.772+0000 7f719c0f8700 1 Processor -- start 2024-03-19T03:08:04.775 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.772+0000 7f719c0f8700 1 -- start start 2024-03-19T03:08:04.775 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.772+0000 7f719c0f8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7194071e40 0x7f719410f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:04.775 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.772+0000 7f719c0f8700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719410f840 con 0x7f7194071e40 2024-03-19T03:08:04.775 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.772+0000 7f7199e94700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7194071e40 0x7f719410f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:04.776 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.772+0000 7f7199e94700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7194071e40 0x7f719410f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.176:60700/0 (socket says 172.21.15.176:60700) 2024-03-19T03:08:04.776 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.772+0000 7f7199e94700 1 -- 172.21.15.176:0/1508053444 learned_addr learned my addr 172.21.15.176:0/1508053444 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:08:04.776 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.773+0000 7f7199e94700 1 -- 172.21.15.176:0/1508053444 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f718c009940 con 0x7f7194071e40 2024-03-19T03:08:04.776 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.773+0000 7f7199e94700 1 --2- 172.21.15.176:0/1508053444 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7194071e40 0x7f719410f300 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f718c0097b0 tx=0x7f718c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:04.777 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.773+0000 7f718a7fc700 1 -- 172.21.15.176:0/1508053444 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f718c012410 con 0x7f7194071e40 2024-03-19T03:08:04.777 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.773+0000 7f719c0f8700 1 -- 172.21.15.176:0/1508053444 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f719410fa40 con 0x7f7194071e40 2024-03-19T03:08:04.777 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.773+0000 7f718a7fc700 1 -- 172.21.15.176:0/1508053444 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f718c012a40 con 0x7f7194071e40 2024-03-19T03:08:04.777 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.773+0000 7f718a7fc700 1 -- 172.21.15.176:0/1508053444 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f718c01b980 con 0x7f7194071e40 2024-03-19T03:08:04.777 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.773+0000 7f719c0f8700 1 -- 172.21.15.176:0/1508053444 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71941105b0 con 0x7f7194071e40 2024-03-19T03:08:04.777 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.774+0000 7f719c0f8700 1 -- 172.21.15.176:0/1508053444 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7194105170 con 0x7f7194071e40 2024-03-19T03:08:04.777 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.774+0000 7f718a7fc700 1 -- 172.21.15.176:0/1508053444 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f718c019040 con 0x7f7194071e40 2024-03-19T03:08:04.778 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.774+0000 7f718a7fc700 1 --2- 172.21.15.176:0/1508053444 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f7180036880 0x7f7180038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:04.778 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.774+0000 7f718a7fc700 1 -- 172.21.15.176:0/1508053444 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f718c049cf0 con 0x7f7194071e40 2024-03-19T03:08:04.778 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.774+0000 7f7199693700 1 --2- 172.21.15.176:0/1508053444 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f7180036880 0x7f7180038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:04.778 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.775+0000 7f7199693700 1 --2- 172.21.15.176:0/1508053444 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f7180036880 0x7f7180038d30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7184009990 tx=0x7f7184006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:04.782 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:04.780+0000 7f718a7fc700 1 -- 172.21.15.176:0/1508053444 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f718c012bb0 con 0x7f7194071e40 2024-03-19T03:08:05.037 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:05.034+0000 7f719c0f8700 1 -- 172.21.15.176:0/1508053444 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f719405f5f0 con 0x7f7194071e40 2024-03-19T03:08:05.037 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:05.035+0000 7f718a7fc700 1 -- 172.21.15.176:0/1508053444 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f718c017090 con 0x7f7194071e40 2024-03-19T03:08:05.038 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T03:08:05.038 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":1,"fsid":"8c8ae530-e59d-11ee-95c9-87774f69a715","modified":"2024-03-19T03:06:12.321211Z","created":"2024-03-19T03:06:12.321211Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-19T03:08:05.040 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:05.037+0000 7f719c0f8700 1 -- 172.21.15.176:0/1508053444 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f7180036880 msgr2=0x7f7180038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:05.040 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:05.037+0000 7f719c0f8700 1 --2- 172.21.15.176:0/1508053444 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f7180036880 0x7f7180038d30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7184009990 tx=0x7f7184006e40 comp rx=0 tx=0).stop 2024-03-19T03:08:05.040 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:05.037+0000 7f719c0f8700 1 -- 172.21.15.176:0/1508053444 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7194071e40 msgr2=0x7f719410f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:05.040 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:05.037+0000 7f719c0f8700 1 --2- 172.21.15.176:0/1508053444 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7194071e40 0x7f719410f300 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f718c0097b0 tx=0x7f718c004d10 comp rx=0 tx=0).stop 2024-03-19T03:08:05.040 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:05.037+0000 7f719c0f8700 1 -- 172.21.15.176:0/1508053444 shutdown_connections 2024-03-19T03:08:05.040 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:05.037+0000 7f719c0f8700 1 --2- 172.21.15.176:0/1508053444 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f7180036880 0x7f7180038d30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:05.040 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:05.037+0000 7f719c0f8700 1 --2- 172.21.15.176:0/1508053444 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7194071e40 0x7f719410f300 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:05.040 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:05.037+0000 7f719c0f8700 1 -- 172.21.15.176:0/1508053444 >> 172.21.15.176:0/1508053444 conn(0x7f71940f8190 msgr2=0x7f7194105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:05.040 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:05.037+0000 7f719c0f8700 1 -- 172.21.15.176:0/1508053444 shutdown_connections 2024-03-19T03:08:05.040 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:05.037+0000 7f719c0f8700 1 -- 172.21.15.176:0/1508053444 wait complete. 2024-03-19T03:08:05.041 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 1 2024-03-19T03:08:06.043 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:05 smithi059 ceph-mon[29133]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:06.043 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:05 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/1508053444' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:06.605 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-19T03:08:06.605 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T03:08:06.858 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:08:06.906 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:08:07.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:07 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:07.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:07 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:07.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:07 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:07.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:07 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:07.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:07 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:07.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:07 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:07.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:07 smithi059 ceph-mon[29133]: Deploying daemon prometheus.smithi059 on smithi059 2024-03-19T03:08:07.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:07 smithi059 ceph-mon[29133]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:08.329 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.325+0000 7facea19f700 1 -- 172.21.15.176:0/1840966851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7face40fca40 msgr2=0x7face40fce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:08.329 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.325+0000 7facea19f700 1 --2- 172.21.15.176:0/1840966851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7face40fca40 0x7face40fce50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7facd40048c0 tx=0x7facd4009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:08.329 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.325+0000 7facea19f700 1 -- 172.21.15.176:0/1840966851 shutdown_connections 2024-03-19T03:08:08.329 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.325+0000 7facea19f700 1 --2- 172.21.15.176:0/1840966851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7face40fca40 0x7face40fce50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:08.329 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.325+0000 7facea19f700 1 -- 172.21.15.176:0/1840966851 >> 172.21.15.176:0/1840966851 conn(0x7face40f81c0 msgr2=0x7face40fa5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:08.329 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.325+0000 7facea19f700 1 -- 172.21.15.176:0/1840966851 shutdown_connections 2024-03-19T03:08:08.329 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.325+0000 7facea19f700 1 -- 172.21.15.176:0/1840966851 wait complete. 2024-03-19T03:08:08.329 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.325+0000 7facea19f700 1 Processor -- start 2024-03-19T03:08:08.329 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.326+0000 7facea19f700 1 -- start start 2024-03-19T03:08:08.330 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.326+0000 7facea19f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7face40fca40 0x7face410d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:08.330 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.326+0000 7facea19f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face410d5d0 con 0x7face40fca40 2024-03-19T03:08:08.330 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.326+0000 7face37fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7face40fca40 0x7face410d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:08.330 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.326+0000 7face37fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7face40fca40 0x7face410d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.176:60710/0 (socket says 172.21.15.176:60710) 2024-03-19T03:08:08.330 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.326+0000 7face37fe700 1 -- 172.21.15.176:0/46721123 learned_addr learned my addr 172.21.15.176:0/46721123 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:08:08.330 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.326+0000 7face37fe700 1 -- 172.21.15.176:0/46721123 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd4005040 con 0x7face40fca40 2024-03-19T03:08:08.330 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.326+0000 7face37fe700 1 --2- 172.21.15.176:0/46721123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7face40fca40 0x7face410d090 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7facd40096b0 tx=0x7facd4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:08.330 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.326+0000 7face0ff9700 1 -- 172.21.15.176:0/46721123 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facd400bed0 con 0x7face40fca40 2024-03-19T03:08:08.330 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.326+0000 7face0ff9700 1 -- 172.21.15.176:0/46721123 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7facd400b5c0 con 0x7face40fca40 2024-03-19T03:08:08.330 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.327+0000 7face0ff9700 1 -- 172.21.15.176:0/46721123 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facd401c5b0 con 0x7face40fca40 2024-03-19T03:08:08.330 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.327+0000 7facea19f700 1 -- 172.21.15.176:0/46721123 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face410d7d0 con 0x7face40fca40 2024-03-19T03:08:08.330 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.327+0000 7facea19f700 1 -- 172.21.15.176:0/46721123 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face410e340 con 0x7face40fca40 2024-03-19T03:08:08.331 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.327+0000 7face0ff9700 1 -- 172.21.15.176:0/46721123 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7facd4013420 con 0x7face40fca40 2024-03-19T03:08:08.332 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.327+0000 7facea19f700 1 -- 172.21.15.176:0/46721123 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face4102f50 con 0x7face40fca40 2024-03-19T03:08:08.332 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.328+0000 7face0ff9700 1 --2- 172.21.15.176:0/46721123 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7facd00368d0 0x7facd0038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:08.332 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.328+0000 7face0ff9700 1 -- 172.21.15.176:0/46721123 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7facd4049a60 con 0x7face40fca40 2024-03-19T03:08:08.332 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.328+0000 7facdaffd700 1 --2- 172.21.15.176:0/46721123 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7facd00368d0 0x7facd0038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:08.332 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.328+0000 7facdaffd700 1 --2- 172.21.15.176:0/46721123 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7facd00368d0 0x7facd0038d80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faccc009990 tx=0x7faccc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:08.336 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.335+0000 7face0ff9700 1 -- 172.21.15.176:0/46721123 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7facd4013a80 con 0x7face40fca40 2024-03-19T03:08:08.589 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.586+0000 7facea19f700 1 -- 172.21.15.176:0/46721123 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7face410ed50 con 0x7face40fca40 2024-03-19T03:08:08.590 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.586+0000 7face0ff9700 1 -- 172.21.15.176:0/46721123 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7facd4011370 con 0x7face40fca40 2024-03-19T03:08:08.590 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T03:08:08.590 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":1,"fsid":"8c8ae530-e59d-11ee-95c9-87774f69a715","modified":"2024-03-19T03:06:12.321211Z","created":"2024-03-19T03:06:12.321211Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-19T03:08:08.591 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.588+0000 7facea19f700 1 -- 172.21.15.176:0/46721123 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7facd00368d0 msgr2=0x7facd0038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:08.591 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.588+0000 7facea19f700 1 --2- 172.21.15.176:0/46721123 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7facd00368d0 0x7facd0038d80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faccc009990 tx=0x7faccc006e40 comp rx=0 tx=0).stop 2024-03-19T03:08:08.591 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.588+0000 7facea19f700 1 -- 172.21.15.176:0/46721123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7face40fca40 msgr2=0x7face410d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:08.591 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.588+0000 7facea19f700 1 --2- 172.21.15.176:0/46721123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7face40fca40 0x7face410d090 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7facd40096b0 tx=0x7facd4009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:08.591 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.588+0000 7facea19f700 1 -- 172.21.15.176:0/46721123 shutdown_connections 2024-03-19T03:08:08.591 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.588+0000 7facea19f700 1 --2- 172.21.15.176:0/46721123 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7facd00368d0 0x7facd0038d80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:08.591 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.588+0000 7facea19f700 1 --2- 172.21.15.176:0/46721123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7face40fca40 0x7face410d090 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:08.591 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.588+0000 7facea19f700 1 -- 172.21.15.176:0/46721123 >> 172.21.15.176:0/46721123 conn(0x7face40f81c0 msgr2=0x7face4103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:08.592 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.589+0000 7facea19f700 1 -- 172.21.15.176:0/46721123 shutdown_connections 2024-03-19T03:08:08.592 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:08.589+0000 7facea19f700 1 -- 172.21.15.176:0/46721123 wait complete. 2024-03-19T03:08:08.592 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 1 2024-03-19T03:08:08.632 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:08 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/46721123' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:09.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:09 smithi059 ceph-mon[29133]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:10.084 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-19T03:08:10.084 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T03:08:10.333 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:08:10.379 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:08:11.814 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.811+0000 7f4a3153f700 1 -- 172.21.15.176:0/3266925328 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a2c0fb950 msgr2=0x7f4a2c0fbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:11.814 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.811+0000 7f4a3153f700 1 --2- 172.21.15.176:0/3266925328 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a2c0fb950 0x7f4a2c0fbd60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c009c60 tx=0x7f4a1c004d10 comp rx=0 tx=0).stop 2024-03-19T03:08:11.814 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.811+0000 7f4a3153f700 1 -- 172.21.15.176:0/3266925328 shutdown_connections 2024-03-19T03:08:11.814 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.811+0000 7f4a3153f700 1 --2- 172.21.15.176:0/3266925328 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a2c0fb950 0x7f4a2c0fbd60 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:11.814 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.811+0000 7f4a3153f700 1 -- 172.21.15.176:0/3266925328 >> 172.21.15.176:0/3266925328 conn(0x7f4a2c0f7090 msgr2=0x7f4a2c0f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:11.814 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.811+0000 7f4a3153f700 1 -- 172.21.15.176:0/3266925328 shutdown_connections 2024-03-19T03:08:11.814 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.811+0000 7f4a3153f700 1 -- 172.21.15.176:0/3266925328 wait complete. 2024-03-19T03:08:11.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.811+0000 7f4a3153f700 1 Processor -- start 2024-03-19T03:08:11.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.812+0000 7f4a3153f700 1 -- start start 2024-03-19T03:08:11.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.812+0000 7f4a3153f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a2c0fb950 0x7f4a2c10d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:11.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.812+0000 7f4a3153f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c10d5b0 con 0x7f4a2c0fb950 2024-03-19T03:08:11.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.812+0000 7f4a2affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a2c0fb950 0x7f4a2c10d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:11.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.812+0000 7f4a2affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a2c0fb950 0x7f4a2c10d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.176:59792/0 (socket says 172.21.15.176:59792) 2024-03-19T03:08:11.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.812+0000 7f4a2affd700 1 -- 172.21.15.176:0/1538617872 learned_addr learned my addr 172.21.15.176:0/1538617872 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:08:11.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.812+0000 7f4a2affd700 1 -- 172.21.15.176:0/1538617872 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1c009940 con 0x7f4a2c0fb950 2024-03-19T03:08:11.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.812+0000 7f4a2affd700 1 --2- 172.21.15.176:0/1538617872 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a2c0fb950 0x7f4a2c10d070 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c004e60 tx=0x7f4a1c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:11.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.813+0000 7f4a23fff700 1 -- 172.21.15.176:0/1538617872 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a1c012410 con 0x7f4a2c0fb950 2024-03-19T03:08:11.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.813+0000 7f4a23fff700 1 -- 172.21.15.176:0/1538617872 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a1c012a40 con 0x7f4a2c0fb950 2024-03-19T03:08:11.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.813+0000 7f4a23fff700 1 -- 172.21.15.176:0/1538617872 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a1c01ba40 con 0x7f4a2c0fb950 2024-03-19T03:08:11.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.813+0000 7f4a3153f700 1 -- 172.21.15.176:0/1538617872 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a2c10d7b0 con 0x7f4a2c0fb950 2024-03-19T03:08:11.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.813+0000 7f4a3153f700 1 -- 172.21.15.176:0/1538617872 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a2c10e320 con 0x7f4a2c0fb950 2024-03-19T03:08:11.816 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.813+0000 7f4a23fff700 1 -- 172.21.15.176:0/1538617872 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f4a1c019040 con 0x7f4a2c0fb950 2024-03-19T03:08:11.816 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.813+0000 7f4a3153f700 1 -- 172.21.15.176:0/1538617872 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a2c101e60 con 0x7f4a2c0fb950 2024-03-19T03:08:11.817 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.813+0000 7f4a23fff700 1 --2- 172.21.15.176:0/1538617872 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f4a180368d0 0x7f4a18038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:11.817 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.814+0000 7f4a23fff700 1 -- 172.21.15.176:0/1538617872 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4a1c049c10 con 0x7f4a2c0fb950 2024-03-19T03:08:11.817 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.814+0000 7f4a227fc700 1 --2- 172.21.15.176:0/1538617872 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f4a180368d0 0x7f4a18038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:11.817 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.814+0000 7f4a227fc700 1 --2- 172.21.15.176:0/1538617872 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f4a180368d0 0x7f4a18038d80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4a14009990 tx=0x7f4a14006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:11.820 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:11.819+0000 7f4a23fff700 1 -- 172.21.15.176:0/1538617872 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f4a1c0227d0 con 0x7f4a2c0fb950 2024-03-19T03:08:12.077 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:12.075+0000 7f4a3153f700 1 -- 172.21.15.176:0/1538617872 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a2c10e560 con 0x7f4a2c0fb950 2024-03-19T03:08:12.077 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:12.075+0000 7f4a23fff700 1 -- 172.21.15.176:0/1538617872 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f4a1c017090 con 0x7f4a2c0fb950 2024-03-19T03:08:12.077 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T03:08:12.077 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":1,"fsid":"8c8ae530-e59d-11ee-95c9-87774f69a715","modified":"2024-03-19T03:06:12.321211Z","created":"2024-03-19T03:06:12.321211Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-19T03:08:12.079 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:12.078+0000 7f4a3153f700 1 -- 172.21.15.176:0/1538617872 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f4a180368d0 msgr2=0x7f4a18038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:12.079 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:12.078+0000 7f4a3153f700 1 --2- 172.21.15.176:0/1538617872 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f4a180368d0 0x7f4a18038d80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4a14009990 tx=0x7f4a14006e40 comp rx=0 tx=0).stop 2024-03-19T03:08:12.079 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:12.078+0000 7f4a3153f700 1 -- 172.21.15.176:0/1538617872 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a2c0fb950 msgr2=0x7f4a2c10d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:12.079 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:12.078+0000 7f4a3153f700 1 --2- 172.21.15.176:0/1538617872 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a2c0fb950 0x7f4a2c10d070 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c004e60 tx=0x7f4a1c004d10 comp rx=0 tx=0).stop 2024-03-19T03:08:12.079 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:12.078+0000 7f4a3153f700 1 -- 172.21.15.176:0/1538617872 shutdown_connections 2024-03-19T03:08:12.079 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:12.078+0000 7f4a3153f700 1 --2- 172.21.15.176:0/1538617872 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f4a180368d0 0x7f4a18038d80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:12.080 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:12.078+0000 7f4a3153f700 1 --2- 172.21.15.176:0/1538617872 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a2c0fb950 0x7f4a2c10d070 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:12.080 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:12.078+0000 7f4a3153f700 1 -- 172.21.15.176:0/1538617872 >> 172.21.15.176:0/1538617872 conn(0x7f4a2c0f7090 msgr2=0x7f4a2c065ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:12.080 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:12.078+0000 7f4a3153f700 1 -- 172.21.15.176:0/1538617872 shutdown_connections 2024-03-19T03:08:12.080 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:12.078+0000 7f4a3153f700 1 -- 172.21.15.176:0/1538617872 wait complete. 2024-03-19T03:08:12.080 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 1 2024-03-19T03:08:12.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:11 smithi059 ceph-mon[29133]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:12.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:11 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:13.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:12 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/1538617872' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:13.645 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-19T03:08:13.645 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T03:08:13.894 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:08:13.939 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:08:14.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:13 smithi059 ceph-mon[29133]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:15.310 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.309+0000 7fbc8de6c700 1 -- 172.21.15.176:0/3293999707 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc880fca60 msgr2=0x7fbc880fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:15.310 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.309+0000 7fbc8de6c700 1 --2- 172.21.15.176:0/3293999707 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc880fca60 0x7fbc880fce70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbc780048c0 tx=0x7fbc78009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:15.310 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.309+0000 7fbc8de6c700 1 -- 172.21.15.176:0/3293999707 shutdown_connections 2024-03-19T03:08:15.310 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.309+0000 7fbc8de6c700 1 --2- 172.21.15.176:0/3293999707 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc880fca60 0x7fbc880fce70 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:15.310 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.309+0000 7fbc8de6c700 1 -- 172.21.15.176:0/3293999707 >> 172.21.15.176:0/3293999707 conn(0x7fbc880f81c0 msgr2=0x7fbc880fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:15.311 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.309+0000 7fbc8de6c700 1 -- 172.21.15.176:0/3293999707 shutdown_connections 2024-03-19T03:08:15.311 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.309+0000 7fbc8de6c700 1 -- 172.21.15.176:0/3293999707 wait complete. 2024-03-19T03:08:15.311 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.309+0000 7fbc8de6c700 1 Processor -- start 2024-03-19T03:08:15.312 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.309+0000 7fbc8de6c700 1 -- start start 2024-03-19T03:08:15.312 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.309+0000 7fbc8de6c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc880fca60 0x7fbc8810d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:15.312 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.309+0000 7fbc8de6c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8810d5d0 con 0x7fbc880fca60 2024-03-19T03:08:15.312 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.310+0000 7fbc877fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc880fca60 0x7fbc8810d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:15.313 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.310+0000 7fbc877fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc880fca60 0x7fbc8810d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.176:59804/0 (socket says 172.21.15.176:59804) 2024-03-19T03:08:15.313 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.310+0000 7fbc877fe700 1 -- 172.21.15.176:0/777446697 learned_addr learned my addr 172.21.15.176:0/777446697 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:08:15.313 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.310+0000 7fbc877fe700 1 -- 172.21.15.176:0/777446697 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc78005040 con 0x7fbc880fca60 2024-03-19T03:08:15.313 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.310+0000 7fbc877fe700 1 --2- 172.21.15.176:0/777446697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc880fca60 0x7fbc8810d090 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbc780096b0 tx=0x7fbc78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:15.313 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.310+0000 7fbc6ffff700 1 -- 172.21.15.176:0/777446697 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc7800bed0 con 0x7fbc880fca60 2024-03-19T03:08:15.313 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.310+0000 7fbc6ffff700 1 -- 172.21.15.176:0/777446697 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbc7800b5c0 con 0x7fbc880fca60 2024-03-19T03:08:15.313 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.310+0000 7fbc8de6c700 1 -- 172.21.15.176:0/777446697 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc8810d7d0 con 0x7fbc880fca60 2024-03-19T03:08:15.313 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.310+0000 7fbc6ffff700 1 -- 172.21.15.176:0/777446697 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc7801c5b0 con 0x7fbc880fca60 2024-03-19T03:08:15.313 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.310+0000 7fbc8de6c700 1 -- 172.21.15.176:0/777446697 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8810e340 con 0x7fbc880fca60 2024-03-19T03:08:15.313 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.311+0000 7fbc6ffff700 1 -- 172.21.15.176:0/777446697 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fbc78013420 con 0x7fbc880fca60 2024-03-19T03:08:15.313 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.311+0000 7fbc8de6c700 1 -- 172.21.15.176:0/777446697 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc88102f70 con 0x7fbc880fca60 2024-03-19T03:08:15.313 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.311+0000 7fbc6ffff700 1 --2- 172.21.15.176:0/777446697 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fbc70036880 0x7fbc70038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:15.313 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.311+0000 7fbc6ffff700 1 -- 172.21.15.176:0/777446697 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbc78049a80 con 0x7fbc880fca60 2024-03-19T03:08:15.314 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.312+0000 7fbc86ffd700 1 --2- 172.21.15.176:0/777446697 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fbc70036880 0x7fbc70038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:15.314 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.312+0000 7fbc86ffd700 1 --2- 172.21.15.176:0/777446697 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fbc70036880 0x7fbc70038d30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c009990 tx=0x7fbc7c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:15.317 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.316+0000 7fbc6ffff700 1 -- 172.21.15.176:0/777446697 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fbc78013a80 con 0x7fbc880fca60 2024-03-19T03:08:15.575 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.573+0000 7fbc8de6c700 1 -- 172.21.15.176:0/777446697 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc8810ed50 con 0x7fbc880fca60 2024-03-19T03:08:15.575 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.574+0000 7fbc6ffff700 1 -- 172.21.15.176:0/777446697 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fbc78011370 con 0x7fbc880fca60 2024-03-19T03:08:15.575 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T03:08:15.575 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":1,"fsid":"8c8ae530-e59d-11ee-95c9-87774f69a715","modified":"2024-03-19T03:06:12.321211Z","created":"2024-03-19T03:06:12.321211Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-19T03:08:15.577 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.576+0000 7fbc8de6c700 1 -- 172.21.15.176:0/777446697 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fbc70036880 msgr2=0x7fbc70038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:15.577 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.576+0000 7fbc8de6c700 1 --2- 172.21.15.176:0/777446697 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fbc70036880 0x7fbc70038d30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c009990 tx=0x7fbc7c006e40 comp rx=0 tx=0).stop 2024-03-19T03:08:15.577 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.576+0000 7fbc8de6c700 1 -- 172.21.15.176:0/777446697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc880fca60 msgr2=0x7fbc8810d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:15.577 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.576+0000 7fbc8de6c700 1 --2- 172.21.15.176:0/777446697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc880fca60 0x7fbc8810d090 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbc780096b0 tx=0x7fbc78009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:15.577 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.576+0000 7fbc8de6c700 1 -- 172.21.15.176:0/777446697 shutdown_connections 2024-03-19T03:08:15.577 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.576+0000 7fbc8de6c700 1 --2- 172.21.15.176:0/777446697 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7fbc70036880 0x7fbc70038d30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:15.578 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.576+0000 7fbc8de6c700 1 --2- 172.21.15.176:0/777446697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc880fca60 0x7fbc8810d090 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:15.578 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.576+0000 7fbc8de6c700 1 -- 172.21.15.176:0/777446697 >> 172.21.15.176:0/777446697 conn(0x7fbc880f81c0 msgr2=0x7fbc88103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:15.578 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.576+0000 7fbc8de6c700 1 -- 172.21.15.176:0/777446697 shutdown_connections 2024-03-19T03:08:15.578 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:15.576+0000 7fbc8de6c700 1 -- 172.21.15.176:0/777446697 wait complete. 2024-03-19T03:08:15.578 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 1 2024-03-19T03:08:16.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:15 smithi059 ceph-mon[29133]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:16.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:15 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/777446697' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:17.149 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-19T03:08:17.149 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T03:08:17.397 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:08:17.444 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:08:18.000 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:17 smithi059 ceph-mon[29133]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:18.896 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.893+0000 7f20b2f7b700 1 -- 172.21.15.176:0/2092613119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20ac0fe0a0 msgr2=0x7f20ac0fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:18.896 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.893+0000 7f20b2f7b700 1 --2- 172.21.15.176:0/2092613119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20ac0fe0a0 0x7f20ac0fe4b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f20a00048c0 tx=0x7f20a0009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:18.896 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.893+0000 7f20b2f7b700 1 -- 172.21.15.176:0/2092613119 shutdown_connections 2024-03-19T03:08:18.896 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.893+0000 7f20b2f7b700 1 --2- 172.21.15.176:0/2092613119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20ac0fe0a0 0x7f20ac0fe4b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:18.896 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.893+0000 7f20b2f7b700 1 -- 172.21.15.176:0/2092613119 >> 172.21.15.176:0/2092613119 conn(0x7f20ac0f6820 msgr2=0x7f20ac0f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:18.896 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.893+0000 7f20b2f7b700 1 -- 172.21.15.176:0/2092613119 shutdown_connections 2024-03-19T03:08:18.896 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.893+0000 7f20b2f7b700 1 -- 172.21.15.176:0/2092613119 wait complete. 2024-03-19T03:08:18.896 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.893+0000 7f20b2f7b700 1 Processor -- start 2024-03-19T03:08:18.896 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.893+0000 7f20b2f7b700 1 -- start start 2024-03-19T03:08:18.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.894+0000 7f20b2f7b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20ac0fe0a0 0x7f20ac10d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:18.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.894+0000 7f20b2f7b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac10d5f0 con 0x7f20ac0fe0a0 2024-03-19T03:08:18.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.894+0000 7f20b0d17700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20ac0fe0a0 0x7f20ac10d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:18.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.894+0000 7f20b0d17700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20ac0fe0a0 0x7f20ac10d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.176:59816/0 (socket says 172.21.15.176:59816) 2024-03-19T03:08:18.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.894+0000 7f20b0d17700 1 -- 172.21.15.176:0/679804096 learned_addr learned my addr 172.21.15.176:0/679804096 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:08:18.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.894+0000 7f20b0d17700 1 -- 172.21.15.176:0/679804096 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a0005040 con 0x7f20ac0fe0a0 2024-03-19T03:08:18.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.894+0000 7f20b0d17700 1 --2- 172.21.15.176:0/679804096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20ac0fe0a0 0x7f20ac10d0b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f20a00096b0 tx=0x7f20a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:18.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.895+0000 7f20a97fa700 1 -- 172.21.15.176:0/679804096 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20a000bed0 con 0x7f20ac0fe0a0 2024-03-19T03:08:18.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.895+0000 7f20b2f7b700 1 -- 172.21.15.176:0/679804096 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ac10d7f0 con 0x7f20ac0fe0a0 2024-03-19T03:08:18.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.895+0000 7f20a97fa700 1 -- 172.21.15.176:0/679804096 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f20a000b5c0 con 0x7f20ac0fe0a0 2024-03-19T03:08:18.898 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.895+0000 7f20a97fa700 1 -- 172.21.15.176:0/679804096 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20a001c5b0 con 0x7f20ac0fe0a0 2024-03-19T03:08:18.898 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.895+0000 7f20b2f7b700 1 -- 172.21.15.176:0/679804096 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ac10e360 con 0x7f20ac0fe0a0 2024-03-19T03:08:18.899 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.896+0000 7f20b2f7b700 1 -- 172.21.15.176:0/679804096 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20ac102f70 con 0x7f20ac0fe0a0 2024-03-19T03:08:18.899 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.896+0000 7f20a97fa700 1 -- 172.21.15.176:0/679804096 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f20a0013a80 con 0x7f20ac0fe0a0 2024-03-19T03:08:18.899 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.896+0000 7f20a97fa700 1 --2- 172.21.15.176:0/679804096 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f209c036880 0x7f209c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:18.899 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.896+0000 7f20a97fa700 1 -- 172.21.15.176:0/679804096 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f20a0049a90 con 0x7f20ac0fe0a0 2024-03-19T03:08:18.900 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.896+0000 7f20abfff700 1 --2- 172.21.15.176:0/679804096 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f209c036880 0x7f209c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:18.900 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.896+0000 7f20abfff700 1 --2- 172.21.15.176:0/679804096 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f209c036880 0x7f209c038d30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2098009990 tx=0x7f2098006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:18.902 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:18.901+0000 7f20a97fa700 1 -- 172.21.15.176:0/679804096 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f20a0011550 con 0x7f20ac0fe0a0 2024-03-19T03:08:19.157 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:19.156+0000 7f20b2f7b700 1 -- 172.21.15.176:0/679804096 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20ac10ed70 con 0x7f20ac0fe0a0 2024-03-19T03:08:19.158 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:19.157+0000 7f20a97fa700 1 -- 172.21.15.176:0/679804096 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f20a0011370 con 0x7f20ac0fe0a0 2024-03-19T03:08:19.158 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T03:08:19.158 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":1,"fsid":"8c8ae530-e59d-11ee-95c9-87774f69a715","modified":"2024-03-19T03:06:12.321211Z","created":"2024-03-19T03:06:12.321211Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-19T03:08:19.160 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:19.159+0000 7f20b2f7b700 1 -- 172.21.15.176:0/679804096 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f209c036880 msgr2=0x7f209c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:19.160 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:19.159+0000 7f20b2f7b700 1 --2- 172.21.15.176:0/679804096 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f209c036880 0x7f209c038d30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2098009990 tx=0x7f2098006e40 comp rx=0 tx=0).stop 2024-03-19T03:08:19.160 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:19.159+0000 7f20b2f7b700 1 -- 172.21.15.176:0/679804096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20ac0fe0a0 msgr2=0x7f20ac10d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:19.160 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:19.159+0000 7f20b2f7b700 1 --2- 172.21.15.176:0/679804096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20ac0fe0a0 0x7f20ac10d0b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f20a00096b0 tx=0x7f20a0009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:19.160 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:19.159+0000 7f20b2f7b700 1 -- 172.21.15.176:0/679804096 shutdown_connections 2024-03-19T03:08:19.160 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:19.159+0000 7f20b2f7b700 1 --2- 172.21.15.176:0/679804096 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f209c036880 0x7f209c038d30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:19.160 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:19.159+0000 7f20b2f7b700 1 --2- 172.21.15.176:0/679804096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20ac0fe0a0 0x7f20ac10d0b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:19.160 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:19.159+0000 7f20b2f7b700 1 -- 172.21.15.176:0/679804096 >> 172.21.15.176:0/679804096 conn(0x7f20ac0f6820 msgr2=0x7f20ac103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:19.161 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:19.159+0000 7f20b2f7b700 1 -- 172.21.15.176:0/679804096 shutdown_connections 2024-03-19T03:08:19.161 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:19.159+0000 7f20b2f7b700 1 -- 172.21.15.176:0/679804096 wait complete. 2024-03-19T03:08:19.161 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 1 2024-03-19T03:08:19.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:19 smithi059 ceph-mon[29133]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:19.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:19 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/679804096' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:20.727 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-19T03:08:20.728 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T03:08:20.976 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:08:21.023 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:08:21.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:21 smithi059 ceph-mon[29133]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:22.309 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.307+0000 7f84b56b3700 1 -- 172.21.15.176:0/2979917404 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84b00fca70 msgr2=0x7f84b00fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:22.309 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.307+0000 7f84b56b3700 1 --2- 172.21.15.176:0/2979917404 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84b00fca70 0x7f84b00fce80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f84a0009c60 tx=0x7f84a0004d10 comp rx=0 tx=0).stop 2024-03-19T03:08:22.309 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.307+0000 7f84b56b3700 1 -- 172.21.15.176:0/2979917404 shutdown_connections 2024-03-19T03:08:22.309 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.307+0000 7f84b56b3700 1 --2- 172.21.15.176:0/2979917404 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84b00fca70 0x7f84b00fce80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:22.309 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.307+0000 7f84b56b3700 1 -- 172.21.15.176:0/2979917404 >> 172.21.15.176:0/2979917404 conn(0x7f84b00f81d0 msgr2=0x7f84b00fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:22.309 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.307+0000 7f84b56b3700 1 -- 172.21.15.176:0/2979917404 shutdown_connections 2024-03-19T03:08:22.309 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.308+0000 7f84b56b3700 1 -- 172.21.15.176:0/2979917404 wait complete. 2024-03-19T03:08:22.311 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.308+0000 7f84b56b3700 1 Processor -- start 2024-03-19T03:08:22.311 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.308+0000 7f84b56b3700 1 -- start start 2024-03-19T03:08:22.311 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.308+0000 7f84b56b3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84b00fca70 0x7f84b010d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:22.311 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.309+0000 7f84b56b3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b010d600 con 0x7f84b00fca70 2024-03-19T03:08:22.311 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.309+0000 7f84aeffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84b00fca70 0x7f84b010d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:22.311 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.309+0000 7f84aeffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84b00fca70 0x7f84b010d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.176:41630/0 (socket says 172.21.15.176:41630) 2024-03-19T03:08:22.311 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.309+0000 7f84aeffd700 1 -- 172.21.15.176:0/2451536641 learned_addr learned my addr 172.21.15.176:0/2451536641 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:08:22.312 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.309+0000 7f84aeffd700 1 -- 172.21.15.176:0/2451536641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a0009940 con 0x7f84b00fca70 2024-03-19T03:08:22.312 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.309+0000 7f84aeffd700 1 --2- 172.21.15.176:0/2451536641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84b00fca70 0x7f84b010d0c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f84a0004e60 tx=0x7f84a0004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:22.312 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.310+0000 7f84a7fff700 1 -- 172.21.15.176:0/2451536641 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84a0012410 con 0x7f84b00fca70 2024-03-19T03:08:22.312 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.310+0000 7f84a7fff700 1 -- 172.21.15.176:0/2451536641 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f84a0012a40 con 0x7f84b00fca70 2024-03-19T03:08:22.312 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.310+0000 7f84a7fff700 1 -- 172.21.15.176:0/2451536641 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84a001ba40 con 0x7f84b00fca70 2024-03-19T03:08:22.312 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.310+0000 7f84b56b3700 1 -- 172.21.15.176:0/2451536641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b010d800 con 0x7f84b00fca70 2024-03-19T03:08:22.312 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.310+0000 7f84b56b3700 1 -- 172.21.15.176:0/2451536641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b010e370 con 0x7f84b00fca70 2024-03-19T03:08:22.313 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.311+0000 7f84a7fff700 1 -- 172.21.15.176:0/2451536641 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f84a0019040 con 0x7f84b00fca70 2024-03-19T03:08:22.313 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.311+0000 7f84b56b3700 1 -- 172.21.15.176:0/2451536641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b0102f80 con 0x7f84b00fca70 2024-03-19T03:08:22.313 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.311+0000 7f84a7fff700 1 --2- 172.21.15.176:0/2451536641 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f849c0368d0 0x7f849c038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:22.313 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.311+0000 7f84a7fff700 1 -- 172.21.15.176:0/2451536641 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f84a0049c10 con 0x7f84b00fca70 2024-03-19T03:08:22.313 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.311+0000 7f84a67fc700 1 --2- 172.21.15.176:0/2451536641 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f849c0368d0 0x7f849c038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:22.313 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.311+0000 7f84a67fc700 1 --2- 172.21.15.176:0/2451536641 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f849c0368d0 0x7f849c038d80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8498009990 tx=0x7f8498006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:22.317 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.316+0000 7f84a7fff700 1 -- 172.21.15.176:0/2451536641 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f84a00227d0 con 0x7f84b00fca70 2024-03-19T03:08:22.573 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.572+0000 7f84b56b3700 1 -- 172.21.15.176:0/2451536641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84b010ed80 con 0x7f84b00fca70 2024-03-19T03:08:22.573 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.572+0000 7f84a7fff700 1 -- 172.21.15.176:0/2451536641 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f84a0017090 con 0x7f84b00fca70 2024-03-19T03:08:22.574 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T03:08:22.574 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":1,"fsid":"8c8ae530-e59d-11ee-95c9-87774f69a715","modified":"2024-03-19T03:06:12.321211Z","created":"2024-03-19T03:06:12.321211Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-19T03:08:22.575 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.574+0000 7f84b56b3700 1 -- 172.21.15.176:0/2451536641 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f849c0368d0 msgr2=0x7f849c038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:22.575 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.574+0000 7f84b56b3700 1 --2- 172.21.15.176:0/2451536641 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f849c0368d0 0x7f849c038d80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8498009990 tx=0x7f8498006e40 comp rx=0 tx=0).stop 2024-03-19T03:08:22.575 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.574+0000 7f84b56b3700 1 -- 172.21.15.176:0/2451536641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84b00fca70 msgr2=0x7f84b010d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:22.576 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.574+0000 7f84b56b3700 1 --2- 172.21.15.176:0/2451536641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84b00fca70 0x7f84b010d0c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f84a0004e60 tx=0x7f84a0004d10 comp rx=0 tx=0).stop 2024-03-19T03:08:22.576 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.574+0000 7f84b56b3700 1 -- 172.21.15.176:0/2451536641 shutdown_connections 2024-03-19T03:08:22.576 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.574+0000 7f84b56b3700 1 --2- 172.21.15.176:0/2451536641 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f849c0368d0 0x7f849c038d80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:22.576 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.574+0000 7f84b56b3700 1 --2- 172.21.15.176:0/2451536641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84b00fca70 0x7f84b010d0c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:22.576 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.574+0000 7f84b56b3700 1 -- 172.21.15.176:0/2451536641 >> 172.21.15.176:0/2451536641 conn(0x7f84b00f81d0 msgr2=0x7f84b0103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:22.576 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.574+0000 7f84b56b3700 1 -- 172.21.15.176:0/2451536641 shutdown_connections 2024-03-19T03:08:22.576 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:22.574+0000 7f84b56b3700 1 -- 172.21.15.176:0/2451536641 wait complete. 2024-03-19T03:08:22.576 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 1 2024-03-19T03:08:23.054 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:22 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/2451536641' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:24.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:23 smithi059 ceph-mon[29133]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:24.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:23 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:24.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:23 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:24.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:23 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:24.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:23 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-19T03:08:24.158 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-19T03:08:24.158 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T03:08:24.418 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:08:24.465 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:08:25.358 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:25 smithi059 ceph-mon[29133]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-19T03:08:25.358 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:25 smithi059 ceph-mon[29133]: mgrmap e14: smithi059.azpecc(active, since 63s) 2024-03-19T03:08:25.821 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.818+0000 7f3b2ca08700 1 -- 172.21.15.176:0/628378725 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b280fb960 msgr2=0x7f3b280fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:25.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.818+0000 7f3b2ca08700 1 --2- 172.21.15.176:0/628378725 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b280fb960 0x7f3b280fbd70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3b180048c0 tx=0x7f3b18009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:25.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.819+0000 7f3b2ca08700 1 -- 172.21.15.176:0/628378725 shutdown_connections 2024-03-19T03:08:25.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.819+0000 7f3b2ca08700 1 --2- 172.21.15.176:0/628378725 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b280fb960 0x7f3b280fbd70 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:25.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.819+0000 7f3b2ca08700 1 -- 172.21.15.176:0/628378725 >> 172.21.15.176:0/628378725 conn(0x7f3b280f70c0 msgr2=0x7f3b280f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:25.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.819+0000 7f3b2ca08700 1 -- 172.21.15.176:0/628378725 shutdown_connections 2024-03-19T03:08:25.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.819+0000 7f3b2ca08700 1 -- 172.21.15.176:0/628378725 wait complete. 2024-03-19T03:08:25.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.819+0000 7f3b2ca08700 1 Processor -- start 2024-03-19T03:08:25.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.819+0000 7f3b2ca08700 1 -- start start 2024-03-19T03:08:25.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.819+0000 7f3b2ca08700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b280fb960 0x7f3b280f9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:25.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.819+0000 7f3b2ca08700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b280f9590 con 0x7f3b280fb960 2024-03-19T03:08:25.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.819+0000 7f3b2659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b280fb960 0x7f3b280f9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:25.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.819+0000 7f3b2659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b280fb960 0x7f3b280f9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.176:41646/0 (socket says 172.21.15.176:41646) 2024-03-19T03:08:25.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.819+0000 7f3b2659c700 1 -- 172.21.15.176:0/2975670755 learned_addr learned my addr 172.21.15.176:0/2975670755 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:08:25.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.820+0000 7f3b2659c700 1 -- 172.21.15.176:0/2975670755 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b18005040 con 0x7f3b280fb960 2024-03-19T03:08:25.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.820+0000 7f3b2659c700 1 --2- 172.21.15.176:0/2975670755 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b280fb960 0x7f3b280f9050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009680 tx=0x7f3b18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:25.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.820+0000 7f3b16ffd700 1 -- 172.21.15.176:0/2975670755 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b1800b5c0 con 0x7f3b280fb960 2024-03-19T03:08:25.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.820+0000 7f3b16ffd700 1 -- 172.21.15.176:0/2975670755 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b1800be70 con 0x7f3b280fb960 2024-03-19T03:08:25.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.820+0000 7f3b2ca08700 1 -- 172.21.15.176:0/2975670755 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b280f7660 con 0x7f3b280fb960 2024-03-19T03:08:25.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.820+0000 7f3b16ffd700 1 -- 172.21.15.176:0/2975670755 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b180135a0 con 0x7f3b280fb960 2024-03-19T03:08:25.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.820+0000 7f3b2ca08700 1 -- 172.21.15.176:0/2975670755 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b280f7b50 con 0x7f3b280fb960 2024-03-19T03:08:25.824 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.821+0000 7f3b16ffd700 1 -- 172.21.15.176:0/2975670755 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 14) v1 ==== 43438+0+0 (secure 0 0 0) 0x7f3b1801c430 con 0x7f3b280fb960 2024-03-19T03:08:25.824 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.821+0000 7f3b2ca08700 1 -- 172.21.15.176:0/2975670755 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b2805f5f0 con 0x7f3b280fb960 2024-03-19T03:08:25.824 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.821+0000 7f3b16ffd700 1 --2- 172.21.15.176:0/2975670755 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f3b100368d0 0x7f3b10038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:25.824 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.821+0000 7f3b16ffd700 1 -- 172.21.15.176:0/2975670755 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3b180494b0 con 0x7f3b280fb960 2024-03-19T03:08:25.824 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.821+0000 7f3b25d9b700 1 -- 172.21.15.176:0/2975670755 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f3b100368d0 msgr2=0x7f3b10038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1857243897 2024-03-19T03:08:25.824 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.821+0000 7f3b25d9b700 1 --2- 172.21.15.176:0/2975670755 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f3b100368d0 0x7f3b10038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T03:08:25.827 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:25.826+0000 7f3b16ffd700 1 -- 172.21.15.176:0/2975670755 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f3b180117e0 con 0x7f3b280fb960 2024-03-19T03:08:26.023 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:26.022+0000 7f3b25d9b700 1 -- 172.21.15.176:0/2975670755 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f3b100368d0 msgr2=0x7f3b10038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1857243897 2024-03-19T03:08:26.023 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:26.022+0000 7f3b25d9b700 1 --2- 172.21.15.176:0/2975670755 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f3b100368d0 0x7f3b10038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T03:08:26.082 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:26.081+0000 7f3b2ca08700 1 -- 172.21.15.176:0/2975670755 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b2804bed0 con 0x7f3b280fb960 2024-03-19T03:08:26.083 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:26.082+0000 7f3b16ffd700 1 -- 172.21.15.176:0/2975670755 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f3b18021070 con 0x7f3b280fb960 2024-03-19T03:08:26.083 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T03:08:26.083 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":1,"fsid":"8c8ae530-e59d-11ee-95c9-87774f69a715","modified":"2024-03-19T03:06:12.321211Z","created":"2024-03-19T03:06:12.321211Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-19T03:08:26.085 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:26.084+0000 7f3b2ca08700 1 -- 172.21.15.176:0/2975670755 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f3b100368d0 msgr2=0x7f3b10038d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T03:08:26.085 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:26.084+0000 7f3b2ca08700 1 --2- 172.21.15.176:0/2975670755 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f3b100368d0 0x7f3b10038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:26.085 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:26.084+0000 7f3b2ca08700 1 -- 172.21.15.176:0/2975670755 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b280fb960 msgr2=0x7f3b280f9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:26.085 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:26.084+0000 7f3b2ca08700 1 --2- 172.21.15.176:0/2975670755 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b280fb960 0x7f3b280f9050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009680 tx=0x7f3b18009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:26.086 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:26.084+0000 7f3b2ca08700 1 -- 172.21.15.176:0/2975670755 shutdown_connections 2024-03-19T03:08:26.086 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:26.084+0000 7f3b2ca08700 1 --2- 172.21.15.176:0/2975670755 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f3b100368d0 0x7f3b10038d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:26.086 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:26.084+0000 7f3b2ca08700 1 --2- 172.21.15.176:0/2975670755 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b280fb960 0x7f3b280f9050 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:26.086 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:26.084+0000 7f3b2ca08700 1 -- 172.21.15.176:0/2975670755 >> 172.21.15.176:0/2975670755 conn(0x7f3b280f70c0 msgr2=0x7f3b28065d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:26.086 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:26.084+0000 7f3b2ca08700 1 -- 172.21.15.176:0/2975670755 shutdown_connections 2024-03-19T03:08:26.086 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:26.084+0000 7f3b2ca08700 1 -- 172.21.15.176:0/2975670755 wait complete. 2024-03-19T03:08:26.086 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 1 2024-03-19T03:08:26.306 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:26 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/2975670755' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:27.652 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-19T03:08:27.652 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T03:08:27.898 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:08:27.946 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:08:29.258 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.255+0000 7f2b84f09700 1 -- 172.21.15.176:0/2544794277 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b800fb950 msgr2=0x7f2b800fbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:29.259 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.255+0000 7f2b84f09700 1 --2- 172.21.15.176:0/2544794277 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b800fb950 0x7f2b800fbd60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2b700048c0 tx=0x7f2b70009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:29.259 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.256+0000 7f2b84f09700 1 -- 172.21.15.176:0/2544794277 shutdown_connections 2024-03-19T03:08:29.259 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.256+0000 7f2b84f09700 1 --2- 172.21.15.176:0/2544794277 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b800fb950 0x7f2b800fbd60 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:29.259 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.256+0000 7f2b84f09700 1 -- 172.21.15.176:0/2544794277 >> 172.21.15.176:0/2544794277 conn(0x7f2b800f7090 msgr2=0x7f2b800f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:29.259 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.256+0000 7f2b84f09700 1 -- 172.21.15.176:0/2544794277 shutdown_connections 2024-03-19T03:08:29.259 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.256+0000 7f2b84f09700 1 -- 172.21.15.176:0/2544794277 wait complete. 2024-03-19T03:08:29.259 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.256+0000 7f2b84f09700 1 Processor -- start 2024-03-19T03:08:29.259 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.256+0000 7f2b84f09700 1 -- start start 2024-03-19T03:08:29.259 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.256+0000 7f2b84f09700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b800fb950 0x7f2b80066840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:29.259 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.256+0000 7f2b84f09700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8010fda0 con 0x7f2b800fb950 2024-03-19T03:08:29.259 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.256+0000 7f2b7e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b800fb950 0x7f2b80066840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:29.260 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.256+0000 7f2b7e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b800fb950 0x7f2b80066840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.176:41666/0 (socket says 172.21.15.176:41666) 2024-03-19T03:08:29.260 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.256+0000 7f2b7e59c700 1 -- 172.21.15.176:0/2750933195 learned_addr learned my addr 172.21.15.176:0/2750933195 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:08:29.260 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.257+0000 7f2b7e59c700 1 -- 172.21.15.176:0/2750933195 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b70005040 con 0x7f2b800fb950 2024-03-19T03:08:29.260 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.257+0000 7f2b7e59c700 1 --2- 172.21.15.176:0/2750933195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b800fb950 0x7f2b80066840 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2b700096b0 tx=0x7f2b70009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:29.260 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.257+0000 7f2b6effd700 1 -- 172.21.15.176:0/2750933195 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b7000bed0 con 0x7f2b800fb950 2024-03-19T03:08:29.260 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.257+0000 7f2b6effd700 1 -- 172.21.15.176:0/2750933195 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b7000b5c0 con 0x7f2b800fb950 2024-03-19T03:08:29.260 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.257+0000 7f2b84f09700 1 -- 172.21.15.176:0/2750933195 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b8010fee0 con 0x7f2b800fb950 2024-03-19T03:08:29.260 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.257+0000 7f2b6effd700 1 -- 172.21.15.176:0/2750933195 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b7001c5b0 con 0x7f2b800fb950 2024-03-19T03:08:29.260 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.257+0000 7f2b84f09700 1 -- 172.21.15.176:0/2750933195 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b801103d0 con 0x7f2b800fb950 2024-03-19T03:08:29.261 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.258+0000 7f2b6effd700 1 -- 172.21.15.176:0/2750933195 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 14) v1 ==== 43438+0+0 (secure 0 0 0) 0x7f2b70013420 con 0x7f2b800fb950 2024-03-19T03:08:29.261 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.258+0000 7f2b84f09700 1 -- 172.21.15.176:0/2750933195 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b80105910 con 0x7f2b800fb950 2024-03-19T03:08:29.261 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.258+0000 7f2b6effd700 1 --2- 172.21.15.176:0/2750933195 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f2b680368d0 0x7f2b68038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:29.261 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.258+0000 7f2b6effd700 1 -- 172.21.15.176:0/2750933195 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2b70049b10 con 0x7f2b800fb950 2024-03-19T03:08:29.261 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.258+0000 7f2b7dd9b700 1 -- 172.21.15.176:0/2750933195 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f2b680368d0 msgr2=0x7f2b68038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1857243897 2024-03-19T03:08:29.262 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.258+0000 7f2b7dd9b700 1 --2- 172.21.15.176:0/2750933195 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f2b680368d0 0x7f2b68038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T03:08:29.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.263+0000 7f2b6effd700 1 -- 172.21.15.176:0/2750933195 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f2b70013a80 con 0x7f2b800fb950 2024-03-19T03:08:29.460 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.459+0000 7f2b7dd9b700 1 -- 172.21.15.176:0/2750933195 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f2b680368d0 msgr2=0x7f2b68038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1857243897 2024-03-19T03:08:29.460 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.459+0000 7f2b7dd9b700 1 --2- 172.21.15.176:0/2750933195 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f2b680368d0 0x7f2b68038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T03:08:29.520 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.519+0000 7f2b84f09700 1 -- 172.21.15.176:0/2750933195 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b8005f5f0 con 0x7f2b800fb950 2024-03-19T03:08:29.521 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.520+0000 7f2b6effd700 1 -- 172.21.15.176:0/2750933195 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f2b70011370 con 0x7f2b800fb950 2024-03-19T03:08:29.521 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T03:08:29.521 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":1,"fsid":"8c8ae530-e59d-11ee-95c9-87774f69a715","modified":"2024-03-19T03:06:12.321211Z","created":"2024-03-19T03:06:12.321211Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-19T03:08:29.523 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.522+0000 7f2b84f09700 1 -- 172.21.15.176:0/2750933195 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f2b680368d0 msgr2=0x7f2b68038d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T03:08:29.523 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.522+0000 7f2b84f09700 1 --2- 172.21.15.176:0/2750933195 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f2b680368d0 0x7f2b68038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:29.523 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.522+0000 7f2b84f09700 1 -- 172.21.15.176:0/2750933195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b800fb950 msgr2=0x7f2b80066840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:29.523 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.522+0000 7f2b84f09700 1 --2- 172.21.15.176:0/2750933195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b800fb950 0x7f2b80066840 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2b700096b0 tx=0x7f2b70009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:29.524 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.522+0000 7f2b84f09700 1 -- 172.21.15.176:0/2750933195 shutdown_connections 2024-03-19T03:08:29.524 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.522+0000 7f2b84f09700 1 --2- 172.21.15.176:0/2750933195 >> [v2:172.21.15.59:6800/1857243897,v1:172.21.15.59:6801/1857243897] conn(0x7f2b680368d0 0x7f2b68038d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:29.524 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.522+0000 7f2b84f09700 1 --2- 172.21.15.176:0/2750933195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b800fb950 0x7f2b80066840 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:29.524 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.522+0000 7f2b84f09700 1 -- 172.21.15.176:0/2750933195 >> 172.21.15.176:0/2750933195 conn(0x7f2b800f7090 msgr2=0x7f2b80102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:29.524 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.522+0000 7f2b84f09700 1 -- 172.21.15.176:0/2750933195 shutdown_connections 2024-03-19T03:08:29.524 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:29.522+0000 7f2b84f09700 1 -- 172.21.15.176:0/2750933195 wait complete. 2024-03-19T03:08:29.524 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 1 2024-03-19T03:08:29.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:29 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/2750933195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:31.089 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-19T03:08:31.090 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T03:08:31.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:31 smithi059 ceph-mon[29133]: Active manager daemon smithi059.azpecc restarted 2024-03-19T03:08:31.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:31 smithi059 ceph-mon[29133]: Activating manager daemon smithi059.azpecc 2024-03-19T03:08:31.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:31 smithi059 ceph-mon[29133]: osdmap e5: 0 total, 0 up, 0 in 2024-03-19T03:08:31.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:31 smithi059 ceph-mon[29133]: mgrmap e15: smithi059.azpecc(active, starting, since 0.00366222s) 2024-03-19T03:08:31.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-03-19T03:08:31.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mgr metadata", "who": "smithi059.azpecc", "id": "smithi059.azpecc"}]: dispatch 2024-03-19T03:08:31.341 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T03:08:31.341 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T03:08:31.341 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T03:08:31.341 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:31 smithi059 ceph-mon[29133]: Manager daemon smithi059.azpecc is now available 2024-03-19T03:08:31.341 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:08:31.341 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:08:31.341 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:31.341 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:08:31.342 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.azpecc/mirror_snapshot_schedule"}]: dispatch 2024-03-19T03:08:31.342 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.azpecc/trash_purge_schedule"}]: dispatch 2024-03-19T03:08:31.359 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:08:31.414 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:08:32.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.942+0000 7f48a19b6700 1 -- 172.21.15.176:0/1874629954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f489c0fe0b0 msgr2=0x7f489c0fe4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:32.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.942+0000 7f48a19b6700 1 --2- 172.21.15.176:0/1874629954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f489c0fe0b0 0x7f489c0fe4c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f48900048c0 tx=0x7f4890009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:32.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.942+0000 7f48a19b6700 1 -- 172.21.15.176:0/1874629954 shutdown_connections 2024-03-19T03:08:32.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.942+0000 7f48a19b6700 1 --2- 172.21.15.176:0/1874629954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f489c0fe0b0 0x7f489c0fe4c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:32.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.942+0000 7f48a19b6700 1 -- 172.21.15.176:0/1874629954 >> 172.21.15.176:0/1874629954 conn(0x7f489c0f6850 msgr2=0x7f489c0f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:32.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.942+0000 7f48a19b6700 1 -- 172.21.15.176:0/1874629954 shutdown_connections 2024-03-19T03:08:32.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.942+0000 7f48a19b6700 1 -- 172.21.15.176:0/1874629954 wait complete. 2024-03-19T03:08:32.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.943+0000 7f48a19b6700 1 Processor -- start 2024-03-19T03:08:32.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.943+0000 7f48a19b6700 1 -- start start 2024-03-19T03:08:32.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.943+0000 7f48a19b6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f489c0fe0b0 0x7f489c10d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:32.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.943+0000 7f48a19b6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c10d600 con 0x7f489c0fe0b0 2024-03-19T03:08:32.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.943+0000 7f489affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f489c0fe0b0 0x7f489c10d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:32.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.943+0000 7f489affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f489c0fe0b0 0x7f489c10d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.176:49966/0 (socket says 172.21.15.176:49966) 2024-03-19T03:08:32.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.943+0000 7f489affd700 1 -- 172.21.15.176:0/2409077989 learned_addr learned my addr 172.21.15.176:0/2409077989 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:08:32.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.943+0000 7f489affd700 1 -- 172.21.15.176:0/2409077989 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4890005040 con 0x7f489c0fe0b0 2024-03-19T03:08:32.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.944+0000 7f489affd700 1 --2- 172.21.15.176:0/2409077989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f489c0fe0b0 0x7f489c10d0c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f48900096b0 tx=0x7f4890009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:32.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.944+0000 7f48837fe700 1 -- 172.21.15.176:0/2409077989 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f489000bed0 con 0x7f489c0fe0b0 2024-03-19T03:08:32.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.944+0000 7f48837fe700 1 -- 172.21.15.176:0/2409077989 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f489000b5c0 con 0x7f489c0fe0b0 2024-03-19T03:08:32.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.944+0000 7f48a19b6700 1 -- 172.21.15.176:0/2409077989 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f489c10d800 con 0x7f489c0fe0b0 2024-03-19T03:08:32.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.944+0000 7f48837fe700 1 -- 172.21.15.176:0/2409077989 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f489001c5b0 con 0x7f489c0fe0b0 2024-03-19T03:08:32.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.944+0000 7f48a19b6700 1 -- 172.21.15.176:0/2409077989 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f489c10e370 con 0x7f489c0fe0b0 2024-03-19T03:08:32.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.945+0000 7f48837fe700 1 -- 172.21.15.176:0/2409077989 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 43289+0+0 (secure 0 0 0) 0x7f489001c710 con 0x7f489c0fe0b0 2024-03-19T03:08:32.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.945+0000 7f48837fe700 1 --2- 172.21.15.176:0/2409077989 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f4884036750 0x7f4884038c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:32.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.945+0000 7f48a19b6700 1 -- 172.21.15.176:0/2409077989 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f489c102f80 con 0x7f489c0fe0b0 2024-03-19T03:08:32.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.945+0000 7f48837fe700 1 -- 172.21.15.176:0/2409077989 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f489004a8c0 con 0x7f489c0fe0b0 2024-03-19T03:08:32.948 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.945+0000 7f489a7fc700 1 --2- 172.21.15.176:0/2409077989 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f4884036750 0x7f4884038c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:32.948 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.946+0000 7f489a7fc700 1 --2- 172.21.15.176:0/2409077989 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f4884036750 0x7f4884038c00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f488c009990 tx=0x7f488c006e40 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:32.951 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:32.950+0000 7f48837fe700 1 -- 172.21.15.176:0/2409077989 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f4890025400 con 0x7f489c0fe0b0 2024-03-19T03:08:33.009 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:32 smithi059 ceph-mon[29133]: [19/Mar/2024:03:08:31] ENGINE Bus STARTING 2024-03-19T03:08:33.009 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:32 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:33.009 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:32 smithi059 ceph-mon[29133]: [19/Mar/2024:03:08:31] ENGINE Serving on https://172.21.15.59:7150 2024-03-19T03:08:33.009 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:32 smithi059 ceph-mon[29133]: [19/Mar/2024:03:08:31] ENGINE Bus STARTED 2024-03-19T03:08:33.009 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:32 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:33.010 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:32 smithi059 ceph-mon[29133]: mgrmap e16: smithi059.azpecc(active, since 1.00579s) 2024-03-19T03:08:33.010 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:32 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:33.220 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:33.219+0000 7f48a19b6700 1 -- 172.21.15.176:0/2409077989 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f489c10ed80 con 0x7f489c0fe0b0 2024-03-19T03:08:33.221 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:33.219+0000 7f48837fe700 1 -- 172.21.15.176:0/2409077989 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f4890048020 con 0x7f489c0fe0b0 2024-03-19T03:08:33.221 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T03:08:33.221 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":1,"fsid":"8c8ae530-e59d-11ee-95c9-87774f69a715","modified":"2024-03-19T03:06:12.321211Z","created":"2024-03-19T03:06:12.321211Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-19T03:08:33.223 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:33.222+0000 7f48a19b6700 1 -- 172.21.15.176:0/2409077989 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f4884036750 msgr2=0x7f4884038c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:33.223 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:33.222+0000 7f48a19b6700 1 --2- 172.21.15.176:0/2409077989 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f4884036750 0x7f4884038c00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f488c009990 tx=0x7f488c006e40 comp rx=0 tx=0).stop 2024-03-19T03:08:33.223 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:33.222+0000 7f48a19b6700 1 -- 172.21.15.176:0/2409077989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f489c0fe0b0 msgr2=0x7f489c10d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:33.223 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:33.222+0000 7f48a19b6700 1 --2- 172.21.15.176:0/2409077989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f489c0fe0b0 0x7f489c10d0c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f48900096b0 tx=0x7f4890009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:33.223 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:33.222+0000 7f48a19b6700 1 -- 172.21.15.176:0/2409077989 shutdown_connections 2024-03-19T03:08:33.223 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:33.222+0000 7f48a19b6700 1 --2- 172.21.15.176:0/2409077989 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f4884036750 0x7f4884038c00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:33.223 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:33.222+0000 7f48a19b6700 1 --2- 172.21.15.176:0/2409077989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f489c0fe0b0 0x7f489c10d0c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:33.223 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:33.222+0000 7f48a19b6700 1 -- 172.21.15.176:0/2409077989 >> 172.21.15.176:0/2409077989 conn(0x7f489c0f6850 msgr2=0x7f489c103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:33.223 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:33.222+0000 7f48a19b6700 1 -- 172.21.15.176:0/2409077989 shutdown_connections 2024-03-19T03:08:33.224 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:33.222+0000 7f48a19b6700 1 -- 172.21.15.176:0/2409077989 wait complete. 2024-03-19T03:08:33.224 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 1 2024-03-19T03:08:34.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:33 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:34.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:33 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/2409077989' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:34.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:33 smithi059 ceph-mon[29133]: mgrmap e17: smithi059.azpecc(active, since 2s) 2024-03-19T03:08:34.814 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-19T03:08:34.814 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T03:08:35.194 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:08:35.259 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-19T03:08:36.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:35 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:36.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:35 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:36.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:35 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:36.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:35 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:36.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:35 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd/host:smithi059", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:08:37.509 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.501+0000 7f04dbf2c700 1 -- 172.21.15.176:0/3808993393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04d4101070 msgr2=0x7f04d4101440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:37.509 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.501+0000 7f04dbf2c700 1 --2- 172.21.15.176:0/3808993393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04d4101070 0x7f04d4101440 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f04d00048c0 tx=0x7f04d0009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:37.509 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.501+0000 7f04dbf2c700 1 -- 172.21.15.176:0/3808993393 shutdown_connections 2024-03-19T03:08:37.509 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.501+0000 7f04dbf2c700 1 --2- 172.21.15.176:0/3808993393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04d4101070 0x7f04d4101440 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:37.510 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.501+0000 7f04dbf2c700 1 -- 172.21.15.176:0/3808993393 >> 172.21.15.176:0/3808993393 conn(0x7f04d40f6c30 msgr2=0x7f04d40f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:37.510 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.501+0000 7f04dbf2c700 1 -- 172.21.15.176:0/3808993393 shutdown_connections 2024-03-19T03:08:37.510 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.501+0000 7f04dbf2c700 1 -- 172.21.15.176:0/3808993393 wait complete. 2024-03-19T03:08:37.510 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.501+0000 7f04dbf2c700 1 Processor -- start 2024-03-19T03:08:37.510 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.501+0000 7f04dbf2c700 1 -- start start 2024-03-19T03:08:37.510 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.502+0000 7f04dbf2c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04d4101070 0x7f04d40fcd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:37.510 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.502+0000 7f04dbf2c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d40fd2c0 con 0x7f04d4101070 2024-03-19T03:08:37.510 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.502+0000 7f04d9cc8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04d4101070 0x7f04d40fcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:37.510 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.502+0000 7f04d9cc8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04d4101070 0x7f04d40fcd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.176:49986/0 (socket says 172.21.15.176:49986) 2024-03-19T03:08:37.510 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.502+0000 7f04d9cc8700 1 -- 172.21.15.176:0/1481123454 learned_addr learned my addr 172.21.15.176:0/1481123454 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:08:37.510 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.502+0000 7f04d9cc8700 1 -- 172.21.15.176:0/1481123454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d0005040 con 0x7f04d4101070 2024-03-19T03:08:37.510 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.502+0000 7f04d9cc8700 1 --2- 172.21.15.176:0/1481123454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04d4101070 0x7f04d40fcd80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f04d000b820 tx=0x7f04d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:37.510 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.502+0000 7f04ca7fc700 1 -- 172.21.15.176:0/1481123454 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04d00093c0 con 0x7f04d4101070 2024-03-19T03:08:37.511 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.502+0000 7f04ca7fc700 1 -- 172.21.15.176:0/1481123454 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f04d0013420 con 0x7f04d4101070 2024-03-19T03:08:37.511 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.502+0000 7f04ca7fc700 1 -- 172.21.15.176:0/1481123454 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04d001c5b0 con 0x7f04d4101070 2024-03-19T03:08:37.511 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.502+0000 7f04dbf2c700 1 -- 172.21.15.176:0/1481123454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d40fd4c0 con 0x7f04d4101070 2024-03-19T03:08:37.511 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.503+0000 7f04dbf2c700 1 -- 172.21.15.176:0/1481123454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d40fd930 con 0x7f04d4101070 2024-03-19T03:08:37.511 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.503+0000 7f04ca7fc700 1 -- 172.21.15.176:0/1481123454 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f04d0009520 con 0x7f04d4101070 2024-03-19T03:08:37.511 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.503+0000 7f04dbf2c700 1 -- 172.21.15.176:0/1481123454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d40f9580 con 0x7f04d4101070 2024-03-19T03:08:37.511 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.503+0000 7f04ca7fc700 1 --2- 172.21.15.176:0/1481123454 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f04c0036970 0x7f04c0038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:37.511 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.504+0000 7f04ca7fc700 1 -- 172.21.15.176:0/1481123454 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f04d0049b10 con 0x7f04d4101070 2024-03-19T03:08:37.511 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.504+0000 7f04d94c7700 1 --2- 172.21.15.176:0/1481123454 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f04c0036970 0x7f04c0038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:37.511 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.504+0000 7f04d94c7700 1 --2- 172.21.15.176:0/1481123454 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f04c0036970 0x7f04c0038e20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f04c4009990 tx=0x7f04c4006e40 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:37.511 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.508+0000 7f04ca7fc700 1 -- 172.21.15.176:0/1481123454 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f04d0013a90 con 0x7f04d4101070 2024-03-19T03:08:37.779 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.777+0000 7f04dbf2c700 1 -- 172.21.15.176:0/1481123454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04d4105400 con 0x7f04d4101070 2024-03-19T03:08:37.780 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.778+0000 7f04ca7fc700 1 -- 172.21.15.176:0/1481123454 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f04d0011370 con 0x7f04d4101070 2024-03-19T03:08:37.780 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T03:08:37.780 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":1,"fsid":"8c8ae530-e59d-11ee-95c9-87774f69a715","modified":"2024-03-19T03:06:12.321211Z","created":"2024-03-19T03:06:12.321211Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-19T03:08:37.783 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.781+0000 7f04dbf2c700 1 -- 172.21.15.176:0/1481123454 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f04c0036970 msgr2=0x7f04c0038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:37.783 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.781+0000 7f04dbf2c700 1 --2- 172.21.15.176:0/1481123454 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f04c0036970 0x7f04c0038e20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f04c4009990 tx=0x7f04c4006e40 comp rx=0 tx=0).stop 2024-03-19T03:08:37.783 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.781+0000 7f04dbf2c700 1 -- 172.21.15.176:0/1481123454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04d4101070 msgr2=0x7f04d40fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:37.783 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.781+0000 7f04dbf2c700 1 --2- 172.21.15.176:0/1481123454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04d4101070 0x7f04d40fcd80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f04d000b820 tx=0x7f04d0009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:37.783 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.781+0000 7f04dbf2c700 1 -- 172.21.15.176:0/1481123454 shutdown_connections 2024-03-19T03:08:37.783 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.781+0000 7f04dbf2c700 1 --2- 172.21.15.176:0/1481123454 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f04c0036970 0x7f04c0038e20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:37.783 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.781+0000 7f04dbf2c700 1 --2- 172.21.15.176:0/1481123454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04d4101070 0x7f04d40fcd80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:37.783 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.781+0000 7f04dbf2c700 1 -- 172.21.15.176:0/1481123454 >> 172.21.15.176:0/1481123454 conn(0x7f04d40f6c30 msgr2=0x7f04d41037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:37.783 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.781+0000 7f04dbf2c700 1 -- 172.21.15.176:0/1481123454 shutdown_connections 2024-03-19T03:08:37.783 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:37.781+0000 7f04dbf2c700 1 -- 172.21.15.176:0/1481123454 wait complete. 2024-03-19T03:08:37.784 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 1 2024-03-19T03:08:37.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:37 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:37.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:37 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:37.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:37 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:37.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:37 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:37.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:37 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:08:37.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:37 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:08:37.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:37 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:08:37.841 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:37 smithi059 ceph-mon[29133]: Updating smithi059:/etc/ceph/ceph.conf 2024-03-19T03:08:37.841 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:37 smithi059 ceph-mon[29133]: Updating smithi176:/etc/ceph/ceph.conf 2024-03-19T03:08:37.841 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:37 smithi059 ceph-mon[29133]: Updating smithi059:/var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:08:37.841 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:37 smithi059 ceph-mon[29133]: Updating smithi176:/var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:08:37.841 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:37 smithi059 ceph-mon[29133]: Updating smithi059:/etc/ceph/ceph.client.admin.keyring 2024-03-19T03:08:37.841 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:37 smithi059 ceph-mon[29133]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-03-19T03:08:38.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:38 smithi059 ceph-mon[29133]: Updating smithi059:/var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T03:08:38.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:38 smithi059 ceph-mon[29133]: Updating smithi176:/var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T03:08:38.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:38.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:38.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:38.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:38.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:38.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi176", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-19T03:08:38.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi176", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-19T03:08:38.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:08:38.841 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:38 smithi059 ceph-mon[29133]: Deploying daemon crash.smithi176 on smithi176 2024-03-19T03:08:38.841 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:38 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/1481123454' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:39.338 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-19T03:08:39.338 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T03:08:39.653 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:08:41.616 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.613+0000 7f80d1b02700 1 -- 172.21.15.176:0/2612676220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80cc102af0 msgr2=0x7f80cc102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:41.616 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.613+0000 7f80d1b02700 1 --2- 172.21.15.176:0/2612676220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80cc102af0 0x7f80cc102ec0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f80c00048c0 tx=0x7f80c0009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:41.616 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.613+0000 7f80d1b02700 1 -- 172.21.15.176:0/2612676220 shutdown_connections 2024-03-19T03:08:41.616 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.613+0000 7f80d1b02700 1 --2- 172.21.15.176:0/2612676220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80cc102af0 0x7f80cc102ec0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:41.616 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.613+0000 7f80d1b02700 1 -- 172.21.15.176:0/2612676220 >> 172.21.15.176:0/2612676220 conn(0x7f80cc0f6c30 msgr2=0x7f80cc0f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:41.617 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.613+0000 7f80d1b02700 1 -- 172.21.15.176:0/2612676220 shutdown_connections 2024-03-19T03:08:41.617 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.613+0000 7f80d1b02700 1 -- 172.21.15.176:0/2612676220 wait complete. 2024-03-19T03:08:41.617 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.613+0000 7f80d1b02700 1 Processor -- start 2024-03-19T03:08:41.617 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.613+0000 7f80d1b02700 1 -- start start 2024-03-19T03:08:41.617 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.614+0000 7f80d1b02700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80cc102af0 0x7f80cc0fc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:41.617 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.614+0000 7f80d1b02700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc0fc5e0 con 0x7f80cc102af0 2024-03-19T03:08:41.617 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.614+0000 7f80cb7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80cc102af0 0x7f80cc0fc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:41.617 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.614+0000 7f80cb7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80cc102af0 0x7f80cc0fc0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.176:57998/0 (socket says 172.21.15.176:57998) 2024-03-19T03:08:41.617 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.614+0000 7f80cb7fe700 1 -- 172.21.15.176:0/3586988509 learned_addr learned my addr 172.21.15.176:0/3586988509 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:08:41.617 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.614+0000 7f80cb7fe700 1 -- 172.21.15.176:0/3586988509 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c0005040 con 0x7f80cc102af0 2024-03-19T03:08:41.617 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.614+0000 7f80cb7fe700 1 --2- 172.21.15.176:0/3586988509 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80cc102af0 0x7f80cc0fc0a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f80c000b7f0 tx=0x7f80c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:41.617 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.615+0000 7f80b3fff700 1 -- 172.21.15.176:0/3586988509 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80c00093f0 con 0x7f80cc102af0 2024-03-19T03:08:41.617 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.615+0000 7f80d1b02700 1 -- 172.21.15.176:0/3586988509 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80cc0fc840 con 0x7f80cc102af0 2024-03-19T03:08:41.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.615+0000 7f80b3fff700 1 -- 172.21.15.176:0/3586988509 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f80c0013420 con 0x7f80cc102af0 2024-03-19T03:08:41.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.615+0000 7f80b3fff700 1 -- 172.21.15.176:0/3586988509 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80c001c590 con 0x7f80cc102af0 2024-03-19T03:08:41.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.615+0000 7f80d1b02700 1 -- 172.21.15.176:0/3586988509 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80cc0fccb0 con 0x7f80cc102af0 2024-03-19T03:08:41.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.616+0000 7f80b3fff700 1 -- 172.21.15.176:0/3586988509 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f80c0009550 con 0x7f80cc102af0 2024-03-19T03:08:41.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.616+0000 7f80d1b02700 1 -- 172.21.15.176:0/3586988509 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80cc0fcef0 con 0x7f80cc102af0 2024-03-19T03:08:41.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.616+0000 7f80b3fff700 1 --2- 172.21.15.176:0/3586988509 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f80b4036920 0x7f80b4038dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:41.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.616+0000 7f80b3fff700 1 -- 172.21.15.176:0/3586988509 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f80c00494a0 con 0x7f80cc102af0 2024-03-19T03:08:41.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.616+0000 7f80caffd700 1 --2- 172.21.15.176:0/3586988509 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f80b4036920 0x7f80b4038dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:41.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.616+0000 7f80caffd700 1 --2- 172.21.15.176:0/3586988509 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f80b4036920 0x7f80b4038dd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f80bc009990 tx=0x7f80bc006e40 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:41.622 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.621+0000 7f80b3fff700 1 -- 172.21.15.176:0/3586988509 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f80c0013590 con 0x7f80cc102af0 2024-03-19T03:08:41.915 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.913+0000 7f80d1b02700 1 -- 172.21.15.176:0/3586988509 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80cc0fd790 con 0x7f80cc102af0 2024-03-19T03:08:41.915 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.914+0000 7f80b3fff700 1 -- 172.21.15.176:0/3586988509 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f80c0021070 con 0x7f80cc102af0 2024-03-19T03:08:41.916 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T03:08:41.916 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":1,"fsid":"8c8ae530-e59d-11ee-95c9-87774f69a715","modified":"2024-03-19T03:06:12.321211Z","created":"2024-03-19T03:06:12.321211Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-19T03:08:41.917 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.916+0000 7f80d1b02700 1 -- 172.21.15.176:0/3586988509 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f80b4036920 msgr2=0x7f80b4038dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:41.917 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.916+0000 7f80d1b02700 1 --2- 172.21.15.176:0/3586988509 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f80b4036920 0x7f80b4038dd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f80bc009990 tx=0x7f80bc006e40 comp rx=0 tx=0).stop 2024-03-19T03:08:41.917 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.916+0000 7f80d1b02700 1 -- 172.21.15.176:0/3586988509 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80cc102af0 msgr2=0x7f80cc0fc0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:41.918 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.916+0000 7f80d1b02700 1 --2- 172.21.15.176:0/3586988509 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80cc102af0 0x7f80cc0fc0a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f80c000b7f0 tx=0x7f80c0009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:41.918 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.916+0000 7f80d1b02700 1 -- 172.21.15.176:0/3586988509 shutdown_connections 2024-03-19T03:08:41.918 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.916+0000 7f80d1b02700 1 --2- 172.21.15.176:0/3586988509 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f80b4036920 0x7f80b4038dd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:41.918 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.916+0000 7f80d1b02700 1 --2- 172.21.15.176:0/3586988509 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80cc102af0 0x7f80cc0fc0a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:41.918 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.916+0000 7f80d1b02700 1 -- 172.21.15.176:0/3586988509 >> 172.21.15.176:0/3586988509 conn(0x7f80cc0f6c30 msgr2=0x7f80cc0f85b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:41.918 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.916+0000 7f80d1b02700 1 -- 172.21.15.176:0/3586988509 shutdown_connections 2024-03-19T03:08:41.918 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:41.916+0000 7f80d1b02700 1 -- 172.21.15.176:0/3586988509 wait complete. 2024-03-19T03:08:41.919 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 1 2024-03-19T03:08:42.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:42 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:42.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:42 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:42.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:42 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:42.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:42 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:42.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:42 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/3586988509' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:43.495 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-19T03:08:43.495 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T03:08:43.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:43 smithi059 ceph-mon[29133]: Deploying daemon node-exporter.smithi176 on smithi176 2024-03-19T03:08:44.343 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:08:46.682 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.679+0000 7f4530f87700 1 -- 172.21.15.176:0/380345067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f452c101070 msgr2=0x7f452c101440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:46.682 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.679+0000 7f4530f87700 1 --2- 172.21.15.176:0/380345067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f452c101070 0x7f452c101440 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f451c0048c0 tx=0x7f451c009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:46.682 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.679+0000 7f4530f87700 1 -- 172.21.15.176:0/380345067 shutdown_connections 2024-03-19T03:08:46.682 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.679+0000 7f4530f87700 1 --2- 172.21.15.176:0/380345067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f452c101070 0x7f452c101440 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:46.682 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.679+0000 7f4530f87700 1 -- 172.21.15.176:0/380345067 >> 172.21.15.176:0/380345067 conn(0x7f452c0f6c30 msgr2=0x7f452c0f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:46.682 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.679+0000 7f4530f87700 1 -- 172.21.15.176:0/380345067 shutdown_connections 2024-03-19T03:08:46.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.679+0000 7f4530f87700 1 -- 172.21.15.176:0/380345067 wait complete. 2024-03-19T03:08:46.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.680+0000 7f4530f87700 1 Processor -- start 2024-03-19T03:08:46.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.680+0000 7f4530f87700 1 -- start start 2024-03-19T03:08:46.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.680+0000 7f4530f87700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f452c101070 0x7f452c1130e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:46.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.680+0000 7f4530f87700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452c113620 con 0x7f452c101070 2024-03-19T03:08:46.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.680+0000 7f452a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f452c101070 0x7f452c1130e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:46.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.680+0000 7f452a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f452c101070 0x7f452c1130e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.176:58008/0 (socket says 172.21.15.176:58008) 2024-03-19T03:08:46.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.680+0000 7f452a59c700 1 -- 172.21.15.176:0/2448606347 learned_addr learned my addr 172.21.15.176:0/2448606347 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:08:46.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.681+0000 7f452a59c700 1 -- 172.21.15.176:0/2448606347 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451c005040 con 0x7f452c101070 2024-03-19T03:08:46.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.681+0000 7f452a59c700 1 --2- 172.21.15.176:0/2448606347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f452c101070 0x7f452c1130e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f451c00b820 tx=0x7f451c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:46.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.683+0000 7f451affd700 1 -- 172.21.15.176:0/2448606347 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f451c0093c0 con 0x7f452c101070 2024-03-19T03:08:46.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.684+0000 7f4530f87700 1 -- 172.21.15.176:0/2448606347 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f452c113820 con 0x7f452c101070 2024-03-19T03:08:46.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.684+0000 7f451affd700 1 -- 172.21.15.176:0/2448606347 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f451c013420 con 0x7f452c101070 2024-03-19T03:08:46.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.684+0000 7f451affd700 1 -- 172.21.15.176:0/2448606347 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f451c01c5b0 con 0x7f452c101070 2024-03-19T03:08:46.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.684+0000 7f4530f87700 1 -- 172.21.15.176:0/2448606347 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f452c113d10 con 0x7f452c101070 2024-03-19T03:08:46.686 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.684+0000 7f451affd700 1 -- 172.21.15.176:0/2448606347 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f451c01c7f0 con 0x7f452c101070 2024-03-19T03:08:46.686 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.685+0000 7f451affd700 1 --2- 172.21.15.176:0/2448606347 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f4514036970 0x7f4514038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:46.687 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.685+0000 7f4530f87700 1 -- 172.21.15.176:0/2448606347 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f452c0fc850 con 0x7f452c101070 2024-03-19T03:08:46.687 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.685+0000 7f451affd700 1 -- 172.21.15.176:0/2448606347 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f451c049ba0 con 0x7f452c101070 2024-03-19T03:08:46.687 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.685+0000 7f4529d9b700 1 --2- 172.21.15.176:0/2448606347 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f4514036970 0x7f4514038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:46.687 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.685+0000 7f4529d9b700 1 --2- 172.21.15.176:0/2448606347 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f4514036970 0x7f4514038e20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4520009990 tx=0x7f4520006e40 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:46.691 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.690+0000 7f451affd700 1 -- 172.21.15.176:0/2448606347 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f451c011d40 con 0x7f452c101070 2024-03-19T03:08:46.956 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.955+0000 7f4530f87700 1 -- 172.21.15.176:0/2448606347 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f452c05f5f0 con 0x7f452c101070 2024-03-19T03:08:46.957 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.955+0000 7f451affd700 1 -- 172.21.15.176:0/2448606347 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f451c024580 con 0x7f452c101070 2024-03-19T03:08:46.957 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T03:08:46.957 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":1,"fsid":"8c8ae530-e59d-11ee-95c9-87774f69a715","modified":"2024-03-19T03:06:12.321211Z","created":"2024-03-19T03:06:12.321211Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-19T03:08:46.959 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.958+0000 7f4530f87700 1 -- 172.21.15.176:0/2448606347 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f4514036970 msgr2=0x7f4514038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:46.959 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.958+0000 7f4530f87700 1 --2- 172.21.15.176:0/2448606347 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f4514036970 0x7f4514038e20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4520009990 tx=0x7f4520006e40 comp rx=0 tx=0).stop 2024-03-19T03:08:46.959 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.958+0000 7f4530f87700 1 -- 172.21.15.176:0/2448606347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f452c101070 msgr2=0x7f452c1130e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:46.960 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.958+0000 7f4530f87700 1 --2- 172.21.15.176:0/2448606347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f452c101070 0x7f452c1130e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f451c00b820 tx=0x7f451c009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:46.960 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.958+0000 7f4530f87700 1 -- 172.21.15.176:0/2448606347 shutdown_connections 2024-03-19T03:08:46.960 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.958+0000 7f4530f87700 1 --2- 172.21.15.176:0/2448606347 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f4514036970 0x7f4514038e20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:46.960 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.958+0000 7f4530f87700 1 --2- 172.21.15.176:0/2448606347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f452c101070 0x7f452c1130e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:46.960 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.958+0000 7f4530f87700 1 -- 172.21.15.176:0/2448606347 >> 172.21.15.176:0/2448606347 conn(0x7f452c0f6c30 msgr2=0x7f452c0f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:46.960 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.958+0000 7f4530f87700 1 -- 172.21.15.176:0/2448606347 shutdown_connections 2024-03-19T03:08:46.960 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:46.958+0000 7f4530f87700 1 -- 172.21.15.176:0/2448606347 wait complete. 2024-03-19T03:08:46.960 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 1 2024-03-19T03:08:47.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:47.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:47.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:47.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:47.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi176.ycmqbw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T03:08:47.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi176.ycmqbw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T03:08:47.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T03:08:47.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:08:47.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:46 smithi059 ceph-mon[29133]: Deploying daemon mgr.smithi176.ycmqbw on smithi176 2024-03-19T03:08:47.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:47.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:08:48.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:47 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/2448606347' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:48.865 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-19T03:08:48.866 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T03:08:49.326 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:08:50.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:50 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:50.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:50 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:50.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:50 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:50.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:50 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:50.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:50 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T03:08:50.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:50 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:08:50.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:50 smithi059 ceph-mon[29133]: Deploying daemon mon.smithi176 on smithi176 2024-03-19T03:08:51.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.417+0000 7f9bcceaa700 1 -- 172.21.15.176:0/3581583937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bc8102a80 msgr2=0x7f9bc8102e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:51.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.417+0000 7f9bcceaa700 1 --2- 172.21.15.176:0/3581583937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bc8102a80 0x7f9bc8102e50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9bb80048c0 tx=0x7f9bb8009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:51.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.417+0000 7f9bcceaa700 1 -- 172.21.15.176:0/3581583937 shutdown_connections 2024-03-19T03:08:51.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.417+0000 7f9bcceaa700 1 --2- 172.21.15.176:0/3581583937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bc8102a80 0x7f9bc8102e50 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:51.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.417+0000 7f9bcceaa700 1 -- 172.21.15.176:0/3581583937 >> 172.21.15.176:0/3581583937 conn(0x7f9bc80f6c30 msgr2=0x7f9bc80f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.417+0000 7f9bcceaa700 1 -- 172.21.15.176:0/3581583937 shutdown_connections 2024-03-19T03:08:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.417+0000 7f9bcceaa700 1 -- 172.21.15.176:0/3581583937 wait complete. 2024-03-19T03:08:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.418+0000 7f9bcceaa700 1 Processor -- start 2024-03-19T03:08:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.418+0000 7f9bcceaa700 1 -- start start 2024-03-19T03:08:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.418+0000 7f9bcceaa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bc8102a80 0x7f9bc81131e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.418+0000 7f9bcceaa700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc8113720 con 0x7f9bc8102a80 2024-03-19T03:08:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.418+0000 7f9bc659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bc8102a80 0x7f9bc81131e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.418+0000 7f9bc659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bc8102a80 0x7f9bc81131e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.176:41992/0 (socket says 172.21.15.176:41992) 2024-03-19T03:08:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.418+0000 7f9bc659c700 1 -- 172.21.15.176:0/1375673765 learned_addr learned my addr 172.21.15.176:0/1375673765 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:08:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.418+0000 7f9bc659c700 1 -- 172.21.15.176:0/1375673765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb8005040 con 0x7f9bc8102a80 2024-03-19T03:08:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.419+0000 7f9bc659c700 1 --2- 172.21.15.176:0/1375673765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bc8102a80 0x7f9bc81131e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9bb800b7f0 tx=0x7f9bb8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:51.422 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.419+0000 7f9bb6ffd700 1 -- 172.21.15.176:0/1375673765 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bb8009370 con 0x7f9bc8102a80 2024-03-19T03:08:51.422 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.419+0000 7f9bcceaa700 1 -- 172.21.15.176:0/1375673765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc8113920 con 0x7f9bc8102a80 2024-03-19T03:08:51.422 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.419+0000 7f9bcceaa700 1 -- 172.21.15.176:0/1375673765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc8113d90 con 0x7f9bc8102a80 2024-03-19T03:08:51.422 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.419+0000 7f9bb6ffd700 1 -- 172.21.15.176:0/1375673765 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9bb8013420 con 0x7f9bc8102a80 2024-03-19T03:08:51.422 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.419+0000 7f9bb6ffd700 1 -- 172.21.15.176:0/1375673765 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bb801c5b0 con 0x7f9bc8102a80 2024-03-19T03:08:51.423 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.420+0000 7f9bcceaa700 1 -- 172.21.15.176:0/1375673765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc80fe200 con 0x7f9bc8102a80 2024-03-19T03:08:51.423 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.420+0000 7f9bb6ffd700 1 -- 172.21.15.176:0/1375673765 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f9bb80094d0 con 0x7f9bc8102a80 2024-03-19T03:08:51.423 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.420+0000 7f9bb6ffd700 1 --2- 172.21.15.176:0/1375673765 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f9bb0036970 0x7f9bb0038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:51.423 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.420+0000 7f9bb6ffd700 1 -- 172.21.15.176:0/1375673765 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f9bb8049ad0 con 0x7f9bc8102a80 2024-03-19T03:08:51.423 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.420+0000 7f9bc5d9b700 1 --2- 172.21.15.176:0/1375673765 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f9bb0036970 0x7f9bb0038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:51.423 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.421+0000 7f9bc5d9b700 1 --2- 172.21.15.176:0/1375673765 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f9bb0036970 0x7f9bb0038e20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc009990 tx=0x7f9bbc006e40 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:51.426 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.425+0000 7f9bb6ffd700 1 -- 172.21.15.176:0/1375673765 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f9bb8013a70 con 0x7f9bc8102a80 2024-03-19T03:08:51.694 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.692+0000 7f9bcceaa700 1 -- 172.21.15.176:0/1375673765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9bc80fe200 con 0x7f9bc8102a80 2024-03-19T03:08:51.694 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.693+0000 7f9bb6ffd700 1 -- 172.21.15.176:0/1375673765 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f9bb8011370 con 0x7f9bc8102a80 2024-03-19T03:08:51.695 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T03:08:51.695 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":1,"fsid":"8c8ae530-e59d-11ee-95c9-87774f69a715","modified":"2024-03-19T03:06:12.321211Z","created":"2024-03-19T03:06:12.321211Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-19T03:08:51.697 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.695+0000 7f9bcceaa700 1 -- 172.21.15.176:0/1375673765 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f9bb0036970 msgr2=0x7f9bb0038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:51.697 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.695+0000 7f9bcceaa700 1 --2- 172.21.15.176:0/1375673765 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f9bb0036970 0x7f9bb0038e20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc009990 tx=0x7f9bbc006e40 comp rx=0 tx=0).stop 2024-03-19T03:08:51.697 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.695+0000 7f9bcceaa700 1 -- 172.21.15.176:0/1375673765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bc8102a80 msgr2=0x7f9bc81131e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:51.697 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.695+0000 7f9bcceaa700 1 --2- 172.21.15.176:0/1375673765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bc8102a80 0x7f9bc81131e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9bb800b7f0 tx=0x7f9bb8009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:51.697 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.695+0000 7f9bcceaa700 1 -- 172.21.15.176:0/1375673765 shutdown_connections 2024-03-19T03:08:51.697 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.695+0000 7f9bcceaa700 1 --2- 172.21.15.176:0/1375673765 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f9bb0036970 0x7f9bb0038e20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:51.697 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.695+0000 7f9bcceaa700 1 --2- 172.21.15.176:0/1375673765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bc8102a80 0x7f9bc81131e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:51.697 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.695+0000 7f9bcceaa700 1 -- 172.21.15.176:0/1375673765 >> 172.21.15.176:0/1375673765 conn(0x7f9bc80f6c30 msgr2=0x7f9bc80f7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:51.697 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.695+0000 7f9bcceaa700 1 -- 172.21.15.176:0/1375673765 shutdown_connections 2024-03-19T03:08:51.697 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:51.695+0000 7f9bcceaa700 1 -- 172.21.15.176:0/1375673765 wait complete. 2024-03-19T03:08:51.698 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 1 2024-03-19T03:08:52.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:51 smithi059 ceph-mon[29133]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:52.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:51 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:52.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:51 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/1375673765' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:53.423 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-19T03:08:53.424 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T03:08:53.793 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi176/config 2024-03-19T03:08:54.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:08:53 smithi059 ceph-mon[29133]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.248+0000 7f09421a8700 1 -- 172.21.15.176:0/1184842473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f093c102ba0 msgr2=0x7f093c102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:55.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.248+0000 7f09421a8700 1 --2- 172.21.15.176:0/1184842473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f093c102ba0 0x7f093c102f70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f092c0048c0 tx=0x7f092c009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:55.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.248+0000 7f09421a8700 1 -- 172.21.15.176:0/1184842473 shutdown_connections 2024-03-19T03:08:55.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.248+0000 7f09421a8700 1 --2- 172.21.15.176:0/1184842473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f093c102ba0 0x7f093c102f70 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:55.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.248+0000 7f09421a8700 1 -- 172.21.15.176:0/1184842473 >> 172.21.15.176:0/1184842473 conn(0x7f093c0f86f0 msgr2=0x7f093c0fab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:55.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.248+0000 7f09421a8700 1 -- 172.21.15.176:0/1184842473 shutdown_connections 2024-03-19T03:08:55.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.248+0000 7f09421a8700 1 -- 172.21.15.176:0/1184842473 wait complete. 2024-03-19T03:08:55.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.249+0000 7f09421a8700 1 Processor -- start 2024-03-19T03:08:55.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.249+0000 7f09421a8700 1 -- start start 2024-03-19T03:08:55.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.249+0000 7f09421a8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f093c102ba0 0x7f093c113240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:55.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.249+0000 7f09421a8700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093c113780 con 0x7f093c102ba0 2024-03-19T03:08:55.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.249+0000 7f093b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f093c102ba0 0x7f093c113240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:55.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.250+0000 7f093b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f093c102ba0 0x7f093c113240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.176:42008/0 (socket says 172.21.15.176:42008) 2024-03-19T03:08:55.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.250+0000 7f093b7fe700 1 -- 172.21.15.176:0/108049165 learned_addr learned my addr 172.21.15.176:0/108049165 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:08:55.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.250+0000 7f093b7fe700 1 -- 172.21.15.176:0/108049165 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092c005040 con 0x7f093c102ba0 2024-03-19T03:08:55.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.250+0000 7f093b7fe700 1 --2- 172.21.15.176:0/108049165 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f093c102ba0 0x7f093c113240 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f092c00b820 tx=0x7f092c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:55.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.250+0000 7f0933fff700 1 -- 172.21.15.176:0/108049165 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f092c0093c0 con 0x7f093c102ba0 2024-03-19T03:08:55.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.250+0000 7f0933fff700 1 -- 172.21.15.176:0/108049165 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f092c013420 con 0x7f093c102ba0 2024-03-19T03:08:55.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.250+0000 7f09421a8700 1 -- 172.21.15.176:0/108049165 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f093c113980 con 0x7f093c102ba0 2024-03-19T03:08:55.258 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.250+0000 7f0933fff700 1 -- 172.21.15.176:0/108049165 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f092c01c5b0 con 0x7f093c102ba0 2024-03-19T03:08:55.258 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.250+0000 7f09421a8700 1 -- 172.21.15.176:0/108049165 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f093c113e70 con 0x7f093c102ba0 2024-03-19T03:08:55.258 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.251+0000 7f0931ffb700 1 -- 172.21.15.176:0/108049165 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09200052b0 con 0x7f093c102ba0 2024-03-19T03:08:55.258 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.252+0000 7f0933fff700 1 -- 172.21.15.176:0/108049165 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f092c01c7f0 con 0x7f093c102ba0 2024-03-19T03:08:55.258 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.252+0000 7f0933fff700 1 --2- 172.21.15.176:0/108049165 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f0928036970 0x7f0928038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:08:55.258 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.252+0000 7f0933fff700 1 -- 172.21.15.176:0/108049165 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f092c049a80 con 0x7f093c102ba0 2024-03-19T03:08:55.259 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.252+0000 7f093affd700 1 --2- 172.21.15.176:0/108049165 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f0928036970 0x7f0928038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:08:55.259 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.252+0000 7f093affd700 1 --2- 172.21.15.176:0/108049165 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f0928036970 0x7f0928038e20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0924009990 tx=0x7f0924006e40 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:08:55.259 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.257+0000 7f0933fff700 1 -- 172.21.15.176:0/108049165 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f092c011370 con 0x7f093c102ba0 2024-03-19T03:08:55.534 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.532+0000 7f0931ffb700 1 -- 172.21.15.176:0/108049165 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0920005560 con 0x7f093c102ba0 2024-03-19T03:08:55.535 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.532+0000 7f0933fff700 1 -- 172.21.15.176:0/108049165 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f092c024b50 con 0x7f093c102ba0 2024-03-19T03:08:55.535 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T03:08:55.536 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":1,"fsid":"8c8ae530-e59d-11ee-95c9-87774f69a715","modified":"2024-03-19T03:06:12.321211Z","created":"2024-03-19T03:06:12.321211Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-19T03:08:55.536 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.535+0000 7f0931ffb700 1 -- 172.21.15.176:0/108049165 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f0928036970 msgr2=0x7f0928038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:55.536 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.535+0000 7f0931ffb700 1 --2- 172.21.15.176:0/108049165 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f0928036970 0x7f0928038e20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0924009990 tx=0x7f0924006e40 comp rx=0 tx=0).stop 2024-03-19T03:08:55.537 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.535+0000 7f0931ffb700 1 -- 172.21.15.176:0/108049165 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f093c102ba0 msgr2=0x7f093c113240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:08:55.537 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.535+0000 7f0931ffb700 1 --2- 172.21.15.176:0/108049165 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f093c102ba0 0x7f093c113240 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f092c00b820 tx=0x7f092c009e30 comp rx=0 tx=0).stop 2024-03-19T03:08:55.537 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.535+0000 7f0931ffb700 1 -- 172.21.15.176:0/108049165 shutdown_connections 2024-03-19T03:08:55.537 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.535+0000 7f0931ffb700 1 --2- 172.21.15.176:0/108049165 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f0928036970 0x7f0928038e20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:55.537 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.535+0000 7f0931ffb700 1 --2- 172.21.15.176:0/108049165 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f093c102ba0 0x7f093c113240 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:08:55.537 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.535+0000 7f0931ffb700 1 -- 172.21.15.176:0/108049165 >> 172.21.15.176:0/108049165 conn(0x7f093c0f86f0 msgr2=0x7f093c0f9dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:08:55.537 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.535+0000 7f0931ffb700 1 -- 172.21.15.176:0/108049165 shutdown_connections 2024-03-19T03:08:55.538 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:08:55.535+0000 7f0931ffb700 1 -- 172.21.15.176:0/108049165 wait complete. 2024-03-19T03:08:55.538 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 1 2024-03-19T03:08:55.626 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-19T03:08:55.626 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-19T03:08:55.626 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-19T03:08:55.626 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-19T03:08:55.626 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-19T03:08:55.626 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-19T03:08:55.626 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-19T03:08:55.626 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-19T03:08:55.627 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-19T03:08:55.627 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-19T03:08:55.627 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-19T03:08:55.627 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-19T03:08:55.627 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.arena_block_size: 4194304 2024-03-19T03:08:55.627 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-19T03:08:55.627 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-19T03:08:55.627 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-19T03:08:55.627 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.disable_auto_compactions: 0 2024-03-19T03:08:55.627 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-19T03:08:55.627 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-19T03:08:55.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-19T03:08:55.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-19T03:08:55.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-19T03:08:55.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-19T03:08:55.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-19T03:08:55.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-19T03:08:55.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-19T03:08:55.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-19T03:08:55.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.table_properties_collectors: 2024-03-19T03:08:55.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.inplace_update_support: 0 2024-03-19T03:08:55.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-19T03:08:55.629 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-19T03:08:55.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-19T03:08:55.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-19T03:08:55.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.bloom_locality: 0 2024-03-19T03:08:55.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.max_successive_merges: 0 2024-03-19T03:08:55.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-19T03:08:55.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.paranoid_file_checks: 0 2024-03-19T03:08:55.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.force_consistency_checks: 1 2024-03-19T03:08:55.880 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.report_bg_io_stats: 0 2024-03-19T03:08:55.880 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.ttl: 2592000 2024-03-19T03:08:55.880 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-19T03:08:55.880 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.enable_blob_files: false 2024-03-19T03:08:55.880 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.min_blob_size: 0 2024-03-19T03:08:55.880 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.blob_file_size: 268435456 2024-03-19T03:08:55.880 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-19T03:08:55.880 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-19T03:08:55.880 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-19T03:08:55.880 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi176/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-19T03:08:55.880 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-03-19T03:08:55.880 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-03-19T03:08:55.880 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710817735623909, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-19T03:08:55.880 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-03-19T03:08:55.881 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-19T03:08:55.881 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710817735624225, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1710817735, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "6420a9cb-37bc-4a15-841b-051c7eb2b035", "db_session_id": "6F8GATLEI48P0N8GJZZI"}} 2024-03-19T03:08:55.881 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-03-19T03:08:55.881 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710817735624523, "job": 1, "event": "recovery_finished"} 2024-03-19T03:08:55.881 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi176/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-19T03:08:55.881 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x5636179b3180 2024-03-19T03:08:55.881 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: DB pointer 0x563617a88000 2024-03-19T03:08:55.881 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-03-19T03:08:55.881 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-19T03:08:55.881 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: ** DB Stats ** 2024-03-19T03:08:55.881 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-19T03:08:55.881 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-19T03:08:55.881 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-19T03:08:55.881 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-19T03:08:55.882 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-19T03:08:55.882 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-19T03:08:55.882 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-19T03:08:55.882 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: 2024-03-19T03:08:55.882 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: ** Compaction Stats [default] ** 2024-03-19T03:08:55.882 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-19T03:08:55.882 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-19T03:08:55.882 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-03-19T03:08:55.882 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-03-19T03:08:55.882 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-03-19T03:08:55.882 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: 2024-03-19T03:08:55.882 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: ** Compaction Stats [default] ** 2024-03-19T03:08:55.882 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-19T03:08:55.882 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-19T03:08:55.883 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-03-19T03:08:55.883 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-19T03:08:55.883 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-19T03:08:55.883 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-19T03:08:55.883 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-19T03:08:55.883 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-19T03:08:55.883 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-19T03:08:55.883 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Cumulative compaction: 0.00 GB write, 0.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T03:08:55.883 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Interval compaction: 0.00 GB write, 0.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T03:08:55.883 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-19T03:08:55.883 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: 2024-03-19T03:08:55.883 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-19T03:08:55.883 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: 2024-03-19T03:08:55.883 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: ** Compaction Stats [default] ** 2024-03-19T03:08:55.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-19T03:08:55.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-19T03:08:55.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-03-19T03:08:55.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-03-19T03:08:55.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-19T03:08:55.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: 2024-03-19T03:08:55.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: ** Compaction Stats [default] ** 2024-03-19T03:08:55.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-19T03:08:55.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-19T03:08:55.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-03-19T03:08:55.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-19T03:08:55.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-19T03:08:55.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-19T03:08:55.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-19T03:08:55.885 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-19T03:08:55.885 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-19T03:08:55.885 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Cumulative compaction: 0.00 GB write, 0.53 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T03:08:55.885 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T03:08:55.885 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-19T03:08:55.885 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: 2024-03-19T03:08:55.885 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-19T03:08:55.885 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: mon.smithi176 does not exist in monmap, will attempt to join an existing cluster 2024-03-19T03:08:55.885 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: using public_addr v2:172.21.15.176:0/0 -> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-03-19T03:08:55.885 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: starting mon.smithi176 rank -1 at public addrs [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] at bind addrs [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi176 fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 2024-03-19T03:08:55.885 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: mon.smithi176@-1(???) e0 preinit fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 2024-03-19T03:08:55.885 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: mon.smithi176@-1(synchronizing).mds e1 new map 2024-03-19T03:08:55.885 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: mon.smithi176@-1(synchronizing).mds e1 print_map 2024-03-19T03:08:55.885 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: e1 2024-03-19T03:08:55.885 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-19T03:08:55.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-19T03:08:55.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: legacy client fscid: -1 2024-03-19T03:08:55.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: 2024-03-19T03:08:55.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout: No filesystems configured 2024-03-19T03:08:55.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: mon.smithi176@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-19T03:08:55.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: mon.smithi176@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-19T03:08:55.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: mon.smithi176@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-19T03:08:55.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: mon.smithi176@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-19T03:08:55.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: mon.smithi176@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-19T03:08:55.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: mon.smithi176@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-19T03:08:55.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: mon.smithi176@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-03-19T03:08:55.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: mon.smithi176@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-19T03:08:55.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: mon.smithi176@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T03:08:55.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: mon.smithi176@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T03:08:55.887 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: mon.smithi176@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T03:08:55.887 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.887 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.887 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.887 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: Added host smithi176 2024-03-19T03:08:55.887 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.887 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.887 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.887 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.887 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.887 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: Deploying daemon alertmanager.smithi059 on smithi059 2024-03-19T03:08:55.887 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T03:08:55.887 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.887 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.888 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.888 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/822620716' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-19T03:08:55.888 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.888 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/822620716' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-19T03:08:55.888 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T03:08:55.888 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.888 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi059:172.21.15.59=smithi059;smithi176:172.21.15.176=smithi176", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:08:55.888 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: Saving service mon spec with placement smithi059:172.21.15.59=smithi059;smithi176:172.21.15.176=smithi176;count:2 2024-03-19T03:08:55.888 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.888 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.888 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.888 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.888 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.888 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.888 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.889 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.889 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-19T03:08:55.889 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-19T03:08:55.889 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.889 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: Deploying daemon grafana.smithi059 on smithi059 2024-03-19T03:08:55.889 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/1732987097' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:55.889 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.889 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.889 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.889 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/1989269426' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:55.889 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.889 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.889 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/280298192' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:55.889 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.890 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.890 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/1508053444' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:55.890 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.891 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.891 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.891 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.891 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.891 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.891 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: Deploying daemon prometheus.smithi059 on smithi059 2024-03-19T03:08:55.891 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.891 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/46721123' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:55.891 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.891 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.891 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.891 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/1538617872' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:55.891 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.891 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.892 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/777446697' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:55.892 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.892 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.892 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/679804096' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:55.892 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.892 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/2451536641' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:55.892 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.892 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.892 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.892 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.892 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-19T03:08:55.892 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14162 172.21.15.59:0/2020386193' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-19T03:08:55.892 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: mgrmap e14: smithi059.azpecc(active, since 63s) 2024-03-19T03:08:55.892 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/2975670755' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:55.893 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/2750933195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:55.893 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: Active manager daemon smithi059.azpecc restarted 2024-03-19T03:08:55.893 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: Activating manager daemon smithi059.azpecc 2024-03-19T03:08:55.893 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: osdmap e5: 0 total, 0 up, 0 in 2024-03-19T03:08:55.893 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: mgrmap e15: smithi059.azpecc(active, starting, since 0.00366222s) 2024-03-19T03:08:55.893 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-03-19T03:08:55.893 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mgr metadata", "who": "smithi059.azpecc", "id": "smithi059.azpecc"}]: dispatch 2024-03-19T03:08:55.893 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T03:08:55.893 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T03:08:55.893 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T03:08:55.893 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: Manager daemon smithi059.azpecc is now available 2024-03-19T03:08:55.893 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:08:55.893 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:08:55.893 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.893 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:08:55.893 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.azpecc/mirror_snapshot_schedule"}]: dispatch 2024-03-19T03:08:55.894 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.azpecc/trash_purge_schedule"}]: dispatch 2024-03-19T03:08:55.894 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: [19/Mar/2024:03:08:31] ENGINE Bus STARTING 2024-03-19T03:08:55.894 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.894 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: [19/Mar/2024:03:08:31] ENGINE Serving on https://172.21.15.59:7150 2024-03-19T03:08:55.894 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: [19/Mar/2024:03:08:31] ENGINE Bus STARTED 2024-03-19T03:08:55.894 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.894 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: mgrmap e16: smithi059.azpecc(active, since 1.00579s) 2024-03-19T03:08:55.894 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.894 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.894 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/2409077989' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:55.894 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: mgrmap e17: smithi059.azpecc(active, since 2s) 2024-03-19T03:08:55.894 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.894 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.894 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.895 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.895 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd/host:smithi059", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:08:55.895 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.895 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.895 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.895 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.895 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:08:55.895 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:08:55.895 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:08:55.895 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: Updating smithi059:/etc/ceph/ceph.conf 2024-03-19T03:08:55.895 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: Updating smithi176:/etc/ceph/ceph.conf 2024-03-19T03:08:55.895 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: Updating smithi059:/var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:08:55.895 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: Updating smithi176:/var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:08:55.895 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: Updating smithi059:/etc/ceph/ceph.client.admin.keyring 2024-03-19T03:08:55.896 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-03-19T03:08:55.896 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: Updating smithi059:/var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T03:08:55.896 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: Updating smithi176:/var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T03:08:55.896 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.896 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.896 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.896 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.896 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.896 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi176", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-19T03:08:55.896 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi176", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-19T03:08:55.896 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:08:55.896 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: Deploying daemon crash.smithi176 on smithi176 2024-03-19T03:08:55.896 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/1481123454' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:55.896 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.896 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.897 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.897 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.897 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/3586988509' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:55.897 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: Deploying daemon node-exporter.smithi176 on smithi176 2024-03-19T03:08:55.897 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.897 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.897 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.897 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.897 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi176.ycmqbw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T03:08:55.897 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi176.ycmqbw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T03:08:55.897 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T03:08:55.897 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:08:55.897 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: Deploying daemon mgr.smithi176.ycmqbw on smithi176 2024-03-19T03:08:55.897 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.898 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:08:55.898 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/2448606347' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:55.898 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.898 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.898 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.898 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.898 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T03:08:55.898 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:08:55.898 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: Deploying daemon mon.smithi176 on smithi176 2024-03-19T03:08:55.898 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.898 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:08:55.898 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/1375673765' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:08:55.898 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:08:55.898 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 ceph-mon[36450]: mon.smithi176@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-19T03:08:55.899 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 bash[36416]: 4130cce6243b4fba49d8fd4c06dbf4390f3471906d8d08de54dc839e9f7a3921 2024-03-19T03:08:55.899 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:08:55 smithi176 systemd[1]: Started Ceph mon.smithi176 for 8c8ae530-e59d-11ee-95c9-87774f69a715. 2024-03-19T03:08:57.282 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-19T03:08:57.283 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T03:08:57.543 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi176/config 2024-03-19T03:09:01.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-03-19T03:09:01.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: mon.smithi059 calling monitor election 2024-03-19T03:09:01.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi176"}]: dispatch 2024-03-19T03:09:01.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: from='mgr.? 172.21.15.176:0/3639665835' entity='mgr.smithi176.ycmqbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi176.ycmqbw/crt"}]: dispatch 2024-03-19T03:09:01.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi176"}]: dispatch 2024-03-19T03:09:01.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:01.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi176"}]: dispatch 2024-03-19T03:09:01.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: mon.smithi176 calling monitor election 2024-03-19T03:09:01.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi176"}]: dispatch 2024-03-19T03:09:01.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:01.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi176"}]: dispatch 2024-03-19T03:09:01.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi176"}]: dispatch 2024-03-19T03:09:01.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: mon.smithi059 is new leader, mons smithi059,smithi176 in quorum (ranks 0,1) 2024-03-19T03:09:01.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: monmap e2: 2 mons at {smithi059=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0],smithi176=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-19T03:09:01.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: fsmap 2024-03-19T03:09:01.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: osdmap e5: 0 total, 0 up, 0 in 2024-03-19T03:09:01.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: mgrmap e17: smithi059.azpecc(active, since 29s) 2024-03-19T03:09:01.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: Standby manager daemon smithi176.ycmqbw started 2024-03-19T03:09:01.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: from='mgr.? 172.21.15.176:0/3639665835' entity='mgr.smithi176.ycmqbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T03:09:01.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: overall HEALTH_OK 2024-03-19T03:09:01.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: from='mgr.? 172.21.15.176:0/3639665835' entity='mgr.smithi176.ycmqbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi176.ycmqbw/key"}]: dispatch 2024-03-19T03:09:01.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:01.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: from='mgr.? 172.21.15.176:0/3639665835' entity='mgr.smithi176.ycmqbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T03:09:01.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:01.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:01.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:00 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:01.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-03-19T03:09:01.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: mon.smithi059 calling monitor election 2024-03-19T03:09:01.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi176"}]: dispatch 2024-03-19T03:09:01.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: from='mgr.? 172.21.15.176:0/3639665835' entity='mgr.smithi176.ycmqbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi176.ycmqbw/crt"}]: dispatch 2024-03-19T03:09:01.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi176"}]: dispatch 2024-03-19T03:09:01.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:01.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi176"}]: dispatch 2024-03-19T03:09:01.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: mon.smithi176 calling monitor election 2024-03-19T03:09:01.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi176"}]: dispatch 2024-03-19T03:09:01.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:01.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi176"}]: dispatch 2024-03-19T03:09:01.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi176"}]: dispatch 2024-03-19T03:09:01.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: mon.smithi059 is new leader, mons smithi059,smithi176 in quorum (ranks 0,1) 2024-03-19T03:09:01.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: monmap e2: 2 mons at {smithi059=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0],smithi176=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-19T03:09:01.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: fsmap 2024-03-19T03:09:01.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: osdmap e5: 0 total, 0 up, 0 in 2024-03-19T03:09:01.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: mgrmap e17: smithi059.azpecc(active, since 29s) 2024-03-19T03:09:01.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: Standby manager daemon smithi176.ycmqbw started 2024-03-19T03:09:01.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: from='mgr.? 172.21.15.176:0/3639665835' entity='mgr.smithi176.ycmqbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T03:09:01.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: overall HEALTH_OK 2024-03-19T03:09:01.130 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: from='mgr.? 172.21.15.176:0/3639665835' entity='mgr.smithi176.ycmqbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi176.ycmqbw/key"}]: dispatch 2024-03-19T03:09:01.130 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:01.130 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: from='mgr.? 172.21.15.176:0/3639665835' entity='mgr.smithi176.ycmqbw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T03:09:01.130 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:01.130 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:01.130 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:00 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:01.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:01 smithi176 ceph-mon[36450]: mgrmap e18: smithi059.azpecc(active, since 29s), standbys: smithi176.ycmqbw 2024-03-19T03:09:01.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:01.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mgr metadata", "who": "smithi176.ycmqbw", "id": "smithi176.ycmqbw"}]: dispatch 2024-03-19T03:09:01.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:09:01.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:01 smithi176 ceph-mon[36450]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:01.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:09:01.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:01.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi176"}]: dispatch 2024-03-19T03:09:02.032 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.028+0000 7fc260eae700 1 -- 172.21.15.176:0/395896657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc248003580 msgr2=0x7fc248005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:09:02.032 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.028+0000 7fc260eae700 1 --2- 172.21.15.176:0/395896657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc248003580 0x7fc248005a00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc25c074200 tx=0x7fc24c00b040 comp rx=0 tx=0).stop 2024-03-19T03:09:02.032 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.028+0000 7fc260eae700 1 -- 172.21.15.176:0/395896657 shutdown_connections 2024-03-19T03:09:02.032 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.028+0000 7fc260eae700 1 --2- 172.21.15.176:0/395896657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc248003580 0x7fc248005a00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:09:02.032 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.028+0000 7fc260eae700 1 -- 172.21.15.176:0/395896657 >> 172.21.15.176:0/395896657 conn(0x7fc25c072030 msgr2=0x7fc25c072430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:09:02.032 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.028+0000 7fc260eae700 1 -- 172.21.15.176:0/395896657 shutdown_connections 2024-03-19T03:09:02.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.028+0000 7fc260eae700 1 -- 172.21.15.176:0/395896657 wait complete. 2024-03-19T03:09:02.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.029+0000 7fc260eae700 1 Processor -- start 2024-03-19T03:09:02.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.029+0000 7fc260eae700 1 -- start start 2024-03-19T03:09:02.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.030+0000 7fc260eae700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc248003580 0x7fc25c113280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:09:02.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.030+0000 7fc260eae700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc25c1137c0 0x7fc25c10d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:09:02.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.030+0000 7fc260eae700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c113dd0 con 0x7fc25c1137c0 2024-03-19T03:09:02.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.030+0000 7fc260eae700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c10d720 con 0x7fc248003580 2024-03-19T03:09:02.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.030+0000 7fc25a59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc248003580 0x7fc25c113280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:09:02.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.030+0000 7fc25a59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc248003580 0x7fc25c113280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:47330/0 (socket says 172.21.15.176:47330) 2024-03-19T03:09:02.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.030+0000 7fc25a59c700 1 -- 172.21.15.176:0/3689024494 learned_addr learned my addr 172.21.15.176:0/3689024494 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:09:02.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.030+0000 7fc251d9b700 1 --2- 172.21.15.176:0/3689024494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc25c1137c0 0x7fc25c10d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:09:02.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.030+0000 7fc25a59c700 1 -- 172.21.15.176:0/3689024494 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc248003580 msgr2=0x7fc25c113280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-19T03:09:02.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.030+0000 7fc25a59c700 1 -- 172.21.15.176:0/3689024494 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc248003580 msgr2=0x7fc25c113280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-19T03:09:02.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.030+0000 7fc25a59c700 1 --2- 172.21.15.176:0/3689024494 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc248003580 0x7fc25c113280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-03-19T03:09:02.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.030+0000 7fc25a59c700 1 --2- 172.21.15.176:0/3689024494 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc248003580 0x7fc25c113280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T03:09:02.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.030+0000 7fc251d9b700 1 -- 172.21.15.176:0/3689024494 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc248003580 msgr2=0x7fc25c113280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T03:09:02.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.030+0000 7fc251d9b700 1 --2- 172.21.15.176:0/3689024494 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc248003580 0x7fc25c113280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:09:02.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.030+0000 7fc251d9b700 1 -- 172.21.15.176:0/3689024494 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc24c005040 con 0x7fc25c1137c0 2024-03-19T03:09:02.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.030+0000 7fc251d9b700 1 --2- 172.21.15.176:0/3689024494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc25c1137c0 0x7fc25c10d1e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc24400b790 tx=0x7fc2440091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:09:02.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.031+0000 7fc2537fe700 1 -- 172.21.15.176:0/3689024494 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc244011810 con 0x7fc25c1137c0 2024-03-19T03:09:02.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.031+0000 7fc2537fe700 1 -- 172.21.15.176:0/3689024494 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc244011e40 con 0x7fc25c1137c0 2024-03-19T03:09:02.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.031+0000 7fc260eae700 1 -- 172.21.15.176:0/3689024494 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc25c10da00 con 0x7fc25c1137c0 2024-03-19T03:09:02.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.031+0000 7fc2537fe700 1 -- 172.21.15.176:0/3689024494 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc24400f2e0 con 0x7fc25c1137c0 2024-03-19T03:09:02.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.031+0000 7fc260eae700 1 -- 172.21.15.176:0/3689024494 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc25c10df40 con 0x7fc25c1137c0 2024-03-19T03:09:02.035 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.032+0000 7fc260eae700 1 -- 172.21.15.176:0/3689024494 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc25c107300 con 0x7fc25c1137c0 2024-03-19T03:09:02.035 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.033+0000 7fc2537fe700 1 -- 172.21.15.176:0/3689024494 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7fc24400f480 con 0x7fc25c1137c0 2024-03-19T03:09:02.035 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.033+0000 7fc2537fe700 1 --2- 172.21.15.176:0/3689024494 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc23c069320 0x7fc23c06b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:09:02.035 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.033+0000 7fc2537fe700 1 -- 172.21.15.176:0/3689024494 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fc244086380 con 0x7fc25c1137c0 2024-03-19T03:09:02.036 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.033+0000 7fc25a59c700 1 --2- 172.21.15.176:0/3689024494 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc23c069320 0x7fc23c06b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:09:02.036 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.034+0000 7fc25a59c700 1 --2- 172.21.15.176:0/3689024494 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc23c069320 0x7fc23c06b7d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc24c007f50 tx=0x7fc24c00b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:09:02.043 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.042+0000 7fc2537fe700 1 -- 172.21.15.176:0/3689024494 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fc2440571a0 con 0x7fc25c1137c0 2024-03-19T03:09:02.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:01 smithi059 ceph-mon[29133]: mgrmap e18: smithi059.azpecc(active, since 29s), standbys: smithi176.ycmqbw 2024-03-19T03:09:02.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:02.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mgr metadata", "who": "smithi176.ycmqbw", "id": "smithi176.ycmqbw"}]: dispatch 2024-03-19T03:09:02.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:09:02.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:01 smithi059 ceph-mon[29133]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:02.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:09:02.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:02.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi176"}]: dispatch 2024-03-19T03:09:02.309 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.307+0000 7fc260eae700 1 -- 172.21.15.176:0/3689024494 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc25c10e8d0 con 0x7fc25c1137c0 2024-03-19T03:09:02.310 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.308+0000 7fc2537fe700 1 -- 172.21.15.176:0/3689024494 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1017 (secure 0 0 0) 0x7fc244015090 con 0x7fc25c1137c0 2024-03-19T03:09:02.310 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-19T03:09:02.310 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":2,"fsid":"8c8ae530-e59d-11ee-95c9-87774f69a715","modified":"2024-03-19T03:08:55.650888Z","created":"2024-03-19T03:06:12.321211Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-19T03:09:02.312 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.310+0000 7fc260eae700 1 -- 172.21.15.176:0/3689024494 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc23c069320 msgr2=0x7fc23c06b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:09:02.312 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.310+0000 7fc260eae700 1 --2- 172.21.15.176:0/3689024494 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc23c069320 0x7fc23c06b7d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc24c007f50 tx=0x7fc24c00b040 comp rx=0 tx=0).stop 2024-03-19T03:09:02.312 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.310+0000 7fc260eae700 1 -- 172.21.15.176:0/3689024494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc25c1137c0 msgr2=0x7fc25c10d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:09:02.312 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.310+0000 7fc260eae700 1 --2- 172.21.15.176:0/3689024494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc25c1137c0 0x7fc25c10d1e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc24400b790 tx=0x7fc2440091a0 comp rx=0 tx=0).stop 2024-03-19T03:09:02.312 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.310+0000 7fc260eae700 1 -- 172.21.15.176:0/3689024494 shutdown_connections 2024-03-19T03:09:02.312 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.310+0000 7fc260eae700 1 --2- 172.21.15.176:0/3689024494 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc23c069320 0x7fc23c06b7d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:09:02.312 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.310+0000 7fc260eae700 1 --2- 172.21.15.176:0/3689024494 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc248003580 0x7fc25c113280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:09:02.312 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.310+0000 7fc260eae700 1 --2- 172.21.15.176:0/3689024494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc25c1137c0 0x7fc25c10d1e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:09:02.312 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.310+0000 7fc260eae700 1 -- 172.21.15.176:0/3689024494 >> 172.21.15.176:0/3689024494 conn(0x7fc25c072030 msgr2=0x7fc25c0fa650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:09:02.313 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.310+0000 7fc260eae700 1 -- 172.21.15.176:0/3689024494 shutdown_connections 2024-03-19T03:09:02.313 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:09:02.310+0000 7fc260eae700 1 -- 172.21.15.176:0/3689024494 wait complete. 2024-03-19T03:09:02.313 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 2 2024-03-19T03:09:03.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:02 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/3689024494' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:09:03.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:02 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/3689024494' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:09:03.195 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-19T03:09:03.196 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-19T03:09:03.479 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:09:03.872 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:03 smithi059 ceph-mon[29133]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:03.872 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:03 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:03.872 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:03 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:03.873 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:03 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:03.873 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:03 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:09:04.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:03 smithi176 ceph-mon[36450]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:04.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:03 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:04.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:03 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:04.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:03 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:04.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:03 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:09:05.225 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:04 smithi059 ceph-mon[29133]: Updating smithi059:/etc/ceph/ceph.conf 2024-03-19T03:09:05.225 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:04 smithi059 ceph-mon[29133]: Updating smithi176:/etc/ceph/ceph.conf 2024-03-19T03:09:05.225 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:04 smithi059 ceph-mon[29133]: Updating smithi176:/var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:09:05.226 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:04 smithi059 ceph-mon[29133]: Updating smithi059:/var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:09:05.226 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:04 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:05.226 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:04 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:05.226 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:04 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:05.226 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:04 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:05.226 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:04 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:05.226 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:04 smithi059 ceph-mon[29133]: Reconfiguring mon.smithi059 (unknown last config time)... 2024-03-19T03:09:05.226 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:04 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T03:09:05.226 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:04 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T03:09:05.226 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:04 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:05.226 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:04 smithi059 ceph-mon[29133]: Reconfiguring daemon mon.smithi059 on smithi059 2024-03-19T03:09:05.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:04 smithi176 ceph-mon[36450]: Updating smithi059:/etc/ceph/ceph.conf 2024-03-19T03:09:05.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:04 smithi176 ceph-mon[36450]: Updating smithi176:/etc/ceph/ceph.conf 2024-03-19T03:09:05.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:04 smithi176 ceph-mon[36450]: Updating smithi176:/var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:09:05.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:04 smithi176 ceph-mon[36450]: Updating smithi059:/var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:09:05.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:04 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:05.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:04 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:05.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:04 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:05.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:04 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:05.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:04 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:05.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:04 smithi176 ceph-mon[36450]: Reconfiguring mon.smithi059 (unknown last config time)... 2024-03-19T03:09:05.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:04 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T03:09:05.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:04 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T03:09:05.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:04 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:05.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:04 smithi176 ceph-mon[36450]: Reconfiguring daemon mon.smithi059 on smithi059 2024-03-19T03:09:06.161 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:06 smithi059 ceph-mon[29133]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:06.161 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:06 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:06.161 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:06 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:06.161 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:06 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi059.azpecc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T03:09:06.162 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:06 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T03:09:06.162 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:06 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:06.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.187+0000 7f319df28700 1 -- 172.21.15.59:0/3316106024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3198102e10 msgr2=0x7f31981031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:09:06.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.187+0000 7f319df28700 1 --2- 172.21.15.59:0/3316106024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3198102e10 0x7f31981031e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3188004940 tx=0x7f3188009e30 comp rx=0 tx=0).stop 2024-03-19T03:09:06.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.187+0000 7f319df28700 1 -- 172.21.15.59:0/3316106024 shutdown_connections 2024-03-19T03:09:06.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.187+0000 7f319df28700 1 --2- 172.21.15.59:0/3316106024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31981037b0 0x7f31980fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:09:06.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.187+0000 7f319df28700 1 --2- 172.21.15.59:0/3316106024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3198102e10 0x7f31981031e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:09:06.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.187+0000 7f319df28700 1 -- 172.21.15.59:0/3316106024 >> 172.21.15.59:0/3316106024 conn(0x7f31980f8980 msgr2=0x7f31980fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:09:06.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.187+0000 7f319df28700 1 -- 172.21.15.59:0/3316106024 shutdown_connections 2024-03-19T03:09:06.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.187+0000 7f319df28700 1 -- 172.21.15.59:0/3316106024 wait complete. 2024-03-19T03:09:06.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.188+0000 7f319df28700 1 Processor -- start 2024-03-19T03:09:06.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.188+0000 7f319df28700 1 -- start start 2024-03-19T03:09:06.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.188+0000 7f319df28700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3198102e10 0x7f3198113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:09:06.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.188+0000 7f319df28700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31981037b0 0x7f31981140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:09:06.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.188+0000 7f319df28700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3198114730 con 0x7f31981037b0 2024-03-19T03:09:06.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.188+0000 7f319df28700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319810db40 con 0x7f3198102e10 2024-03-19T03:09:06.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.188+0000 7f3196ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31981037b0 0x7f31981140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:09:06.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.188+0000 7f3196ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31981037b0 0x7f31981140e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44346/0 (socket says 172.21.15.59:44346) 2024-03-19T03:09:06.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.188+0000 7f3196ffd700 1 -- 172.21.15.59:0/780246859 learned_addr learned my addr 172.21.15.59:0/780246859 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:09:06.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.188+0000 7f31977fe700 1 --2- 172.21.15.59:0/780246859 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3198102e10 0x7f3198113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:09:06.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.189+0000 7f3196ffd700 1 -- 172.21.15.59:0/780246859 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3198102e10 msgr2=0x7f3198113ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:09:06.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.189+0000 7f3196ffd700 1 --2- 172.21.15.59:0/780246859 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3198102e10 0x7f3198113ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:09:06.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.189+0000 7f3196ffd700 1 -- 172.21.15.59:0/780246859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3188005040 con 0x7f31981037b0 2024-03-19T03:09:06.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.189+0000 7f3196ffd700 1 --2- 172.21.15.59:0/780246859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31981037b0 0x7f31981140e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3180002a50 tx=0x7f3180004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:09:06.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.189+0000 7f318ffff700 1 -- 172.21.15.59:0/780246859 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3180007cc0 con 0x7f31981037b0 2024-03-19T03:09:06.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.189+0000 7f318ffff700 1 -- 172.21.15.59:0/780246859 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f318000f040 con 0x7f31981037b0 2024-03-19T03:09:06.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.189+0000 7f319df28700 1 -- 172.21.15.59:0/780246859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f319810de20 con 0x7f31981037b0 2024-03-19T03:09:06.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.189+0000 7f318ffff700 1 -- 172.21.15.59:0/780246859 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f318000d9c0 con 0x7f31981037b0 2024-03-19T03:09:06.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.189+0000 7f319df28700 1 -- 172.21.15.59:0/780246859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f319810e3e0 con 0x7f31981037b0 2024-03-19T03:09:06.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.190+0000 7f319df28700 1 -- 172.21.15.59:0/780246859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3198100120 con 0x7f31981037b0 2024-03-19T03:09:06.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.191+0000 7f318ffff700 1 -- 172.21.15.59:0/780246859 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f3180009730 con 0x7f31981037b0 2024-03-19T03:09:06.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.191+0000 7f318ffff700 1 --2- 172.21.15.59:0/780246859 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f3184069210 0x7f318406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:09:06.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.191+0000 7f318ffff700 1 -- 172.21.15.59:0/780246859 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f3180085d00 con 0x7f31981037b0 2024-03-19T03:09:06.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.191+0000 7f31977fe700 1 --2- 172.21.15.59:0/780246859 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f3184069210 0x7f318406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:09:06.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.191+0000 7f31977fe700 1 --2- 172.21.15.59:0/780246859 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f3184069210 0x7f318406b6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3188004f90 tx=0x7f3188009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:09:06.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.196+0000 7f318ffff700 1 -- 172.21.15.59:0/780246859 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f3180053110 con 0x7f31981037b0 2024-03-19T03:09:06.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:06 smithi176 ceph-mon[36450]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:06.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:06 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:06.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:06 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:06.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:06 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi059.azpecc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T03:09:06.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:06 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T03:09:06.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:06 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:06.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.394+0000 7f319df28700 1 -- 172.21.15.59:0/780246859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f319810f1f0 con 0x7f31981037b0 2024-03-19T03:09:06.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.395+0000 7f318ffff700 1 -- 172.21.15.59:0/780246859 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7f31800854b0 con 0x7f31981037b0 2024-03-19T03:09:06.397 INFO:teuthology.orchestra.run.smithi059.stdout:# minimal ceph.conf for 8c8ae530-e59d-11ee-95c9-87774f69a715 2024-03-19T03:09:06.397 INFO:teuthology.orchestra.run.smithi059.stdout:[global] 2024-03-19T03:09:06.397 INFO:teuthology.orchestra.run.smithi059.stdout: fsid = 8c8ae530-e59d-11ee-95c9-87774f69a715 2024-03-19T03:09:06.397 INFO:teuthology.orchestra.run.smithi059.stdout: mon_host = [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-03-19T03:09:06.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.396+0000 7f319df28700 1 -- 172.21.15.59:0/780246859 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f3184069210 msgr2=0x7f318406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:09:06.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.396+0000 7f319df28700 1 --2- 172.21.15.59:0/780246859 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f3184069210 0x7f318406b6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3188004f90 tx=0x7f3188009e30 comp rx=0 tx=0).stop 2024-03-19T03:09:06.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.396+0000 7f319df28700 1 -- 172.21.15.59:0/780246859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31981037b0 msgr2=0x7f31981140e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:09:06.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.396+0000 7f319df28700 1 --2- 172.21.15.59:0/780246859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31981037b0 0x7f31981140e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3180002a50 tx=0x7f3180004600 comp rx=0 tx=0).stop 2024-03-19T03:09:06.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.397+0000 7f319df28700 1 -- 172.21.15.59:0/780246859 shutdown_connections 2024-03-19T03:09:06.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.397+0000 7f319df28700 1 --2- 172.21.15.59:0/780246859 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f3184069210 0x7f318406b6c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:09:06.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.397+0000 7f319df28700 1 --2- 172.21.15.59:0/780246859 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3198102e10 0x7f3198113ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:09:06.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.397+0000 7f319df28700 1 --2- 172.21.15.59:0/780246859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31981037b0 0x7f31981140e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:09:06.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.397+0000 7f319df28700 1 -- 172.21.15.59:0/780246859 >> 172.21.15.59:0/780246859 conn(0x7f31980f8980 msgr2=0x7f3198107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:09:06.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.397+0000 7f319df28700 1 -- 172.21.15.59:0/780246859 shutdown_connections 2024-03-19T03:09:06.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:06.397+0000 7f319df28700 1 -- 172.21.15.59:0/780246859 wait complete. 2024-03-19T03:09:07.205 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-19T03:09:07.205 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-19T03:09:07.205 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.conf 2024-03-19T03:09:07.238 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-19T03:09:07.238 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T03:09:07.308 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-19T03:09:07.308 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.conf 2024-03-19T03:09:07.343 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-19T03:09:07.343 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T03:09:07.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:07 smithi176 ceph-mon[36450]: Reconfiguring mgr.smithi059.azpecc (unknown last config time)... 2024-03-19T03:09:07.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:07 smithi176 ceph-mon[36450]: Reconfiguring daemon mgr.smithi059.azpecc on smithi059 2024-03-19T03:09:07.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:07 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/780246859' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:07.408 INFO:tasks.cephadm:Deploying OSDs... 2024-03-19T03:09:07.408 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-19T03:09:07.408 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2024-03-19T03:09:07.425 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-19T03:09:07.425 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_1 2024-03-19T03:09:07.481 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-19T03:09:07.482 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T03:09:07.482 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-19T03:09:07.482 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:09:07.482 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T03:09:07.482 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-03-19 03:08:19.243304159 +0000 2024-03-19T03:09:07.482 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-03-19 02:49:32.068427058 +0000 2024-03-19T03:09:07.482 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-03-19 02:49:32.068427058 +0000 2024-03-19T03:09:07.482 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-03-19 02:49:32.068427058 +0000 2024-03-19T03:09:07.482 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-19T03:09:07.546 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:07 smithi059 ceph-mon[29133]: Reconfiguring mgr.smithi059.azpecc (unknown last config time)... 2024-03-19T03:09:07.546 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:07 smithi059 ceph-mon[29133]: Reconfiguring daemon mgr.smithi059.azpecc on smithi059 2024-03-19T03:09:07.546 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:07 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/780246859' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:07.549 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-03-19T03:09:07.549 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-03-19T03:09:07.549 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000265844 s, 1.9 MB/s 2024-03-19T03:09:07.550 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-19T03:09:07.606 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_2 2024-03-19T03:09:07.661 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-19T03:09:07.661 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T03:09:07.661 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-19T03:09:07.662 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:09:07.662 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T03:09:07.662 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-03-19 03:08:19.243304159 +0000 2024-03-19T03:09:07.662 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-03-19 02:49:32.398422501 +0000 2024-03-19T03:09:07.662 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-03-19 02:49:32.398422501 +0000 2024-03-19T03:09:07.662 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-03-19 02:49:32.398422501 +0000 2024-03-19T03:09:07.662 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-19T03:09:07.727 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-03-19T03:09:07.728 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-03-19T03:09:07.728 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000185063 s, 2.8 MB/s 2024-03-19T03:09:07.729 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-19T03:09:07.786 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_3 2024-03-19T03:09:07.841 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-19T03:09:07.841 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T03:09:07.841 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-19T03:09:07.841 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:09:07.841 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T03:09:07.841 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-03-19 03:08:19.244304145 +0000 2024-03-19T03:09:07.841 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-03-19 02:49:32.735417848 +0000 2024-03-19T03:09:07.842 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-03-19 02:49:32.735417848 +0000 2024-03-19T03:09:07.842 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-03-19 02:49:32.735417848 +0000 2024-03-19T03:09:07.842 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-19T03:09:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-03-19T03:09:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-03-19T03:09:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000179914 s, 2.8 MB/s 2024-03-19T03:09:07.908 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-19T03:09:07.966 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_4 2024-03-19T03:09:08.021 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-19T03:09:08.021 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T03:09:08.021 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-19T03:09:08.022 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:09:08.022 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T03:09:08.022 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-03-19 03:08:19.244304145 +0000 2024-03-19T03:09:08.022 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-03-19 02:49:33.073413181 +0000 2024-03-19T03:09:08.022 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-03-19 02:49:33.073413181 +0000 2024-03-19T03:09:08.022 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-03-19 02:49:33.073413181 +0000 2024-03-19T03:09:08.022 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-19T03:09:08.088 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-03-19T03:09:08.088 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-03-19T03:09:08.088 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.00020688 s, 2.5 MB/s 2024-03-19T03:09:08.089 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-19T03:09:08.144 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-19T03:09:08.144 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2024-03-19T03:09:08.163 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-19T03:09:08.163 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_1 2024-03-19T03:09:08.221 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-19T03:09:08.221 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T03:09:08.221 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-19T03:09:08.221 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:09:08.221 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T03:09:08.221 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-19 03:08:53.717377013 +0000 2024-03-19T03:09:08.221 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-19 02:40:47.241732386 +0000 2024-03-19T03:09:08.221 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-19 02:40:47.241732386 +0000 2024-03-19T03:09:08.221 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-19 02:40:47.241732386 +0000 2024-03-19T03:09:08.222 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-19T03:09:08.290 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-19T03:09:08.290 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-19T03:09:08.290 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000234737 s, 2.2 MB/s 2024-03-19T03:09:08.291 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-19T03:09:08.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:08 smithi059 ceph-mon[29133]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:08.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:08 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:08.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:08 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:08.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:08 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi059", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-19T03:09:08.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:08 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:08.349 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_2 2024-03-19T03:09:08.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:08 smithi176 ceph-mon[36450]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:08.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:08 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:08.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:08 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:08.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:08 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi059", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-19T03:09:08.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:08 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:08.394 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-19T03:09:08.394 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T03:09:08.394 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-19T03:09:08.394 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:09:08.394 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T03:09:08.394 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-19 03:08:53.718377003 +0000 2024-03-19T03:09:08.394 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-19 02:40:47.584729249 +0000 2024-03-19T03:09:08.394 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-19 02:40:47.584729249 +0000 2024-03-19T03:09:08.394 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-19 02:40:47.584729249 +0000 2024-03-19T03:09:08.395 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-19T03:09:08.462 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-19T03:09:08.462 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-19T03:09:08.462 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.00020581 s, 2.5 MB/s 2024-03-19T03:09:08.463 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-19T03:09:08.520 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_3 2024-03-19T03:09:08.576 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-19T03:09:08.576 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T03:09:08.576 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-19T03:09:08.576 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:09:08.576 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T03:09:08.576 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-19 03:08:53.719376994 +0000 2024-03-19T03:09:08.576 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-19 02:40:47.941725983 +0000 2024-03-19T03:09:08.577 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-19 02:40:47.941725983 +0000 2024-03-19T03:09:08.577 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-19 02:40:47.941725983 +0000 2024-03-19T03:09:08.577 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-19T03:09:08.643 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-19T03:09:08.643 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-19T03:09:08.643 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000208317 s, 2.5 MB/s 2024-03-19T03:09:08.645 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-19T03:09:08.701 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_4 2024-03-19T03:09:08.758 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-19T03:09:08.758 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T03:09:08.758 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-19T03:09:08.758 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:09:08.758 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T03:09:08.758 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-19 03:08:53.719376994 +0000 2024-03-19T03:09:08.758 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-19 02:40:48.277722910 +0000 2024-03-19T03:09:08.758 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-19 02:40:48.277722910 +0000 2024-03-19T03:09:08.758 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-19 02:40:48.277722910 +0000 2024-03-19T03:09:08.759 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-19T03:09:08.824 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-19T03:09:08.825 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-19T03:09:08.825 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000187955 s, 2.7 MB/s 2024-03-19T03:09:08.826 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-19T03:09:08.882 INFO:tasks.cephadm:Deploying osd.0 on smithi059 with /dev/vg_nvme/lv_4... 2024-03-19T03:09:08.882 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-19T03:09:09.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:09 smithi059 ceph-mon[29133]: Reconfiguring crash.smithi059 (monmap changed)... 2024-03-19T03:09:09.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:09 smithi059 ceph-mon[29133]: Reconfiguring daemon crash.smithi059 on smithi059 2024-03-19T03:09:09.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:09 smithi176 ceph-mon[36450]: Reconfiguring crash.smithi059 (monmap changed)... 2024-03-19T03:09:09.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:09 smithi176 ceph-mon[36450]: Reconfiguring daemon crash.smithi059 on smithi059 2024-03-19T03:09:09.478 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:09:10.175 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:10 smithi059 ceph-mon[29133]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:10.175 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:10 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:10.176 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:10 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:10.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:10 smithi176 ceph-mon[36450]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:10.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:10 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:10.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:10 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:11.409 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:11 smithi059 ceph-mon[29133]: Reconfiguring alertmanager.smithi059 (dependencies changed)... 2024-03-19T03:09:11.409 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:11 smithi059 ceph-mon[29133]: Reconfiguring daemon alertmanager.smithi059 on smithi059 2024-03-19T03:09:11.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:11 smithi176 ceph-mon[36450]: Reconfiguring alertmanager.smithi059 (dependencies changed)... 2024-03-19T03:09:11.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:11 smithi176 ceph-mon[36450]: Reconfiguring daemon alertmanager.smithi059 on smithi059 2024-03-19T03:09:12.281 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:12 smithi059 ceph-mon[29133]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:12.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:12 smithi176 ceph-mon[36450]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:12.758 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-19T03:09:12.758 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-19T03:09:12.758 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10+0 records in 2024-03-19T03:09:12.758 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2024-03-19T03:09:12.759 INFO:teuthology.orchestra.run.smithi059.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0191589 s, 547 MB/s 2024-03-19T03:09:12.759 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 2024-03-19T03:09:12.759 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping successful for: 2024-03-19T03:09:13.655 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi059:vg_nvme/lv_4 2024-03-19T03:09:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:09:14.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:14 smithi059 ceph-mon[29133]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:14.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:14 smithi176 ceph-mon[36450]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:15.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.917+0000 7faa523fc700 1 -- 172.21.15.59:0/1505217212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa4c066f50 msgr2=0x7faa4c101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:09:15.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.917+0000 7faa523fc700 1 --2- 172.21.15.59:0/1505217212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa4c066f50 0x7faa4c101dc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7faa3c004940 tx=0x7faa3c009e30 comp rx=0 tx=0).stop 2024-03-19T03:09:15.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.917+0000 7faa523fc700 1 -- 172.21.15.59:0/1505217212 shutdown_connections 2024-03-19T03:09:15.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.917+0000 7faa523fc700 1 --2- 172.21.15.59:0/1505217212 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faa4c102300 0x7faa4c1046e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:09:15.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.917+0000 7faa523fc700 1 --2- 172.21.15.59:0/1505217212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa4c066f50 0x7faa4c101dc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:09:15.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.917+0000 7faa523fc700 1 -- 172.21.15.59:0/1505217212 >> 172.21.15.59:0/1505217212 conn(0x7faa4c0f7450 msgr2=0x7faa4c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:09:15.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.917+0000 7faa523fc700 1 -- 172.21.15.59:0/1505217212 shutdown_connections 2024-03-19T03:09:15.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.917+0000 7faa523fc700 1 -- 172.21.15.59:0/1505217212 wait complete. 2024-03-19T03:09:15.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.917+0000 7faa523fc700 1 Processor -- start 2024-03-19T03:09:15.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.917+0000 7faa523fc700 1 -- start start 2024-03-19T03:09:15.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.918+0000 7faa523fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa4c066f50 0x7faa4c10fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:09:15.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.918+0000 7faa523fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faa4c102300 0x7faa4c1100c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:09:15.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.918+0000 7faa523fc700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4c1106c0 con 0x7faa4c066f50 2024-03-19T03:09:15.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.918+0000 7faa523fc700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4c110830 con 0x7faa4c102300 2024-03-19T03:09:15.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.918+0000 7faa4bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa4c066f50 0x7faa4c10fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:09:15.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.918+0000 7faa4bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa4c066f50 0x7faa4c10fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54630/0 (socket says 172.21.15.59:54630) 2024-03-19T03:09:15.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.918+0000 7faa4bfff700 1 -- 172.21.15.59:0/1864649593 learned_addr learned my addr 172.21.15.59:0/1864649593 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:09:15.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.918+0000 7faa4b7fe700 1 --2- 172.21.15.59:0/1864649593 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faa4c102300 0x7faa4c1100c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:09:15.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.918+0000 7faa4bfff700 1 -- 172.21.15.59:0/1864649593 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faa4c102300 msgr2=0x7faa4c1100c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:09:15.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.918+0000 7faa4bfff700 1 --2- 172.21.15.59:0/1864649593 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faa4c102300 0x7faa4c1100c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:09:15.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.918+0000 7faa4bfff700 1 -- 172.21.15.59:0/1864649593 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3c005040 con 0x7faa4c066f50 2024-03-19T03:09:15.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.918+0000 7faa4bfff700 1 --2- 172.21.15.59:0/1864649593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa4c066f50 0x7faa4c10fb80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00b970 tx=0x7faa3c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:09:15.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.918+0000 7faa48ff9700 1 -- 172.21.15.59:0/1864649593 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa3c00e040 con 0x7faa4c066f50 2024-03-19T03:09:15.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.918+0000 7faa523fc700 1 -- 172.21.15.59:0/1864649593 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa4c115230 con 0x7faa4c066f50 2024-03-19T03:09:15.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.918+0000 7faa48ff9700 1 -- 172.21.15.59:0/1864649593 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa3c00bbf0 con 0x7faa4c066f50 2024-03-19T03:09:15.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.918+0000 7faa48ff9700 1 -- 172.21.15.59:0/1864649593 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa3c01c630 con 0x7faa4c066f50 2024-03-19T03:09:15.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.918+0000 7faa523fc700 1 -- 172.21.15.59:0/1864649593 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa4c1157f0 con 0x7faa4c066f50 2024-03-19T03:09:15.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.919+0000 7faa523fc700 1 -- 172.21.15.59:0/1864649593 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa4c04c710 con 0x7faa4c066f50 2024-03-19T03:09:15.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.920+0000 7faa48ff9700 1 -- 172.21.15.59:0/1864649593 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7faa3c00bd60 con 0x7faa4c066f50 2024-03-19T03:09:15.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.920+0000 7faa48ff9700 1 --2- 172.21.15.59:0/1864649593 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7faa38069210 0x7faa3806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:09:15.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.920+0000 7faa48ff9700 1 -- 172.21.15.59:0/1864649593 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7faa3c087440 con 0x7faa4c066f50 2024-03-19T03:09:15.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.920+0000 7faa4b7fe700 1 --2- 172.21.15.59:0/1864649593 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7faa38069210 0x7faa3806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:09:15.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.921+0000 7faa4b7fe700 1 --2- 172.21.15.59:0/1864649593 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7faa38069210 0x7faa3806b6c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7faa34007fd0 tx=0x7faa34009040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:09:15.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:15.925+0000 7faa48ff9700 1 -- 172.21.15.59:0/1864649593 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7faa3c058390 con 0x7faa4c066f50 2024-03-19T03:09:16.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:16.135+0000 7faa523fc700 1 -- 172.21.15.59:0/1864649593 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7faa4c0f8f40 con 0x7faa38069210 2024-03-19T03:09:16.413 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:16 smithi059 ceph-mon[29133]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:16.413 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:16 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:09:16.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:16 smithi176 ceph-mon[36450]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:16.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:16 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:09:17.310 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:17 smithi059 ceph-mon[29133]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:09:17.311 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:17 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:09:17.311 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:17 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:09:17.311 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:17 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:17.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:17 smithi176 ceph-mon[36450]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:09:17.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:17 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:09:17.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:17 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:09:17.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:17 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:18.182 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:18 smithi059 ceph-mon[29133]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:18.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:18 smithi176 ceph-mon[36450]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:19.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:19 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:19.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:19 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:19.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:19 smithi059 ceph-mon[29133]: Reconfiguring grafana.smithi059 (dependencies changed)... 2024-03-19T03:09:19.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:19 smithi059 ceph-mon[29133]: Reconfiguring daemon grafana.smithi059 on smithi059 2024-03-19T03:09:19.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:19 smithi059 ceph-mon[29133]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:19.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:19 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:19.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:19 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:19.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:19 smithi176 ceph-mon[36450]: Reconfiguring grafana.smithi059 (dependencies changed)... 2024-03-19T03:09:19.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:19 smithi176 ceph-mon[36450]: Reconfiguring daemon grafana.smithi059 on smithi059 2024-03-19T03:09:19.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:19 smithi176 ceph-mon[36450]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:20.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:20 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/1069237315' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6a12d03-3dd9-40d4-b54e-4b2041c18e48"}]: dispatch 2024-03-19T03:09:20.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:20 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/1069237315' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6a12d03-3dd9-40d4-b54e-4b2041c18e48"}]': finished 2024-03-19T03:09:20.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:20 smithi059 ceph-mon[29133]: osdmap e6: 1 total, 0 up, 1 in 2024-03-19T03:09:20.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:20 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:09:20.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:20 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/986237243' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:09:20.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:20 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/1069237315' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6a12d03-3dd9-40d4-b54e-4b2041c18e48"}]: dispatch 2024-03-19T03:09:20.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:20 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/1069237315' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6a12d03-3dd9-40d4-b54e-4b2041c18e48"}]': finished 2024-03-19T03:09:20.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:20 smithi176 ceph-mon[36450]: osdmap e6: 1 total, 0 up, 1 in 2024-03-19T03:09:20.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:20 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:09:20.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:20 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/986237243' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:09:21.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:21 smithi059 ceph-mon[29133]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:21.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:21 smithi176 ceph-mon[36450]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:24.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:23 smithi059 ceph-mon[29133]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:24.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:23 smithi176 ceph-mon[36450]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:26.355 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:26 smithi059 ceph-mon[29133]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:26.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:26 smithi176 ceph-mon[36450]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:28.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:28 smithi176 ceph-mon[36450]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:28.456 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:28 smithi059 ceph-mon[29133]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:29.887 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:29 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:29.888 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:29 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:29.888 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:29 smithi059 ceph-mon[29133]: Reconfiguring prometheus.smithi059 (dependencies changed)... 2024-03-19T03:09:29.888 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:29 smithi059 ceph-mon[29133]: Reconfiguring daemon prometheus.smithi059 on smithi059 2024-03-19T03:09:29.888 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:29 smithi059 ceph-mon[29133]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:30.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:29 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:30.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:29 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:30.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:29 smithi176 ceph-mon[36450]: Reconfiguring prometheus.smithi059 (dependencies changed)... 2024-03-19T03:09:30.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:29 smithi176 ceph-mon[36450]: Reconfiguring daemon prometheus.smithi059 on smithi059 2024-03-19T03:09:30.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:29 smithi176 ceph-mon[36450]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:31.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:31 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-19T03:09:31.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:31 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:31.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:31 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:09:31.432 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-19T03:09:31.432 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:31.432 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:09:32.331 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:32 smithi059 ceph-mon[29133]: Deploying daemon osd.0 on smithi059 2024-03-19T03:09:32.331 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:32 smithi059 ceph-mon[29133]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:32.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:32 smithi176 ceph-mon[36450]: Deploying daemon osd.0 on smithi059 2024-03-19T03:09:32.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:32 smithi176 ceph-mon[36450]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:34.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:34 smithi176 ceph-mon[36450]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:34.379 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:34 smithi059 ceph-mon[29133]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:35.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:35 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:35.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:35 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:35.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:35 smithi059 ceph-mon[29133]: Reconfiguring crash.smithi176 (monmap changed)... 2024-03-19T03:09:35.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:35 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi176", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-19T03:09:35.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:35 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:35.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:35 smithi059 ceph-mon[29133]: Reconfiguring daemon crash.smithi176 on smithi176 2024-03-19T03:09:35.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:35 smithi059 ceph-mon[29133]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:35.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:35 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:35.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:35 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:35.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:35 smithi176 ceph-mon[36450]: Reconfiguring crash.smithi176 (monmap changed)... 2024-03-19T03:09:35.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:35 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi176", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-19T03:09:35.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:35 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:35.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:35 smithi176 ceph-mon[36450]: Reconfiguring daemon crash.smithi176 on smithi176 2024-03-19T03:09:35.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:35 smithi176 ceph-mon[36450]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:37.234 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:36 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:37.234 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:36 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:37.234 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:36 smithi059 ceph-mon[29133]: Reconfiguring mgr.smithi176.ycmqbw (monmap changed)... 2024-03-19T03:09:37.234 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:36 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi176.ycmqbw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T03:09:37.235 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:36 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T03:09:37.235 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:36 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:37.235 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:36 smithi059 ceph-mon[29133]: Reconfiguring daemon mgr.smithi176.ycmqbw on smithi176 2024-03-19T03:09:37.271 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:36 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:37.271 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:36 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:37.271 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:36 smithi176 ceph-mon[36450]: Reconfiguring mgr.smithi176.ycmqbw (monmap changed)... 2024-03-19T03:09:37.271 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:36 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi176.ycmqbw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T03:09:37.271 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:36 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T03:09:37.271 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:36 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:37.271 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:36 smithi176 ceph-mon[36450]: Reconfiguring daemon mgr.smithi176.ycmqbw on smithi176 2024-03-19T03:09:38.031 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:37 smithi176 ceph-mon[36450]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:38.031 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:37 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:38.031 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:37 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:38.031 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:37 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T03:09:38.031 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:37 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T03:09:38.031 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:37 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:38.330 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:37 smithi059 ceph-mon[29133]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:38.330 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:37 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:38.330 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:37 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:38.330 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:37 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T03:09:38.330 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:37 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T03:09:38.330 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:37 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:39.261 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:38 smithi176 ceph-mon[36450]: Reconfiguring mon.smithi176 (monmap changed)... 2024-03-19T03:09:39.261 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:38 smithi176 ceph-mon[36450]: Reconfiguring daemon mon.smithi176 on smithi176 2024-03-19T03:09:39.261 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:39.261 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:39.261 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-19T03:09:39.261 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi059.front.sepia.ceph.com:9093"}]: dispatch 2024-03-19T03:09:39.261 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:39.261 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-19T03:09:39.261 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi059.front.sepia.ceph.com:3000"}]: dispatch 2024-03-19T03:09:39.261 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:39.261 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-19T03:09:39.261 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi059.front.sepia.ceph.com:9095"}]: dispatch 2024-03-19T03:09:39.262 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:39.262 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:09:39.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:38 smithi059 ceph-mon[29133]: Reconfiguring mon.smithi176 (monmap changed)... 2024-03-19T03:09:39.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:38 smithi059 ceph-mon[29133]: Reconfiguring daemon mon.smithi176 on smithi176 2024-03-19T03:09:39.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:39.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:39.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-19T03:09:39.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi059.front.sepia.ceph.com:9093"}]: dispatch 2024-03-19T03:09:39.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:39.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-19T03:09:39.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi059.front.sepia.ceph.com:3000"}]: dispatch 2024-03-19T03:09:39.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:39.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-19T03:09:39.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi059.front.sepia.ceph.com:9095"}]: dispatch 2024-03-19T03:09:39.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:39.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:09:40.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:39 smithi059 ceph-mon[29133]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-19T03:09:40.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:39 smithi059 ceph-mon[29133]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi059.front.sepia.ceph.com:9093"}]: dispatch 2024-03-19T03:09:40.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:39 smithi059 ceph-mon[29133]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-19T03:09:40.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:39 smithi059 ceph-mon[29133]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi059.front.sepia.ceph.com:3000"}]: dispatch 2024-03-19T03:09:40.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:39 smithi059 ceph-mon[29133]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-19T03:09:40.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:39 smithi059 ceph-mon[29133]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi059.front.sepia.ceph.com:9095"}]: dispatch 2024-03-19T03:09:40.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:39 smithi059 ceph-mon[29133]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:40.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:39 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:40.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:39 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:40.306 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:39 smithi176 ceph-mon[36450]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-19T03:09:40.306 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:39 smithi176 ceph-mon[36450]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi059.front.sepia.ceph.com:9093"}]: dispatch 2024-03-19T03:09:40.306 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:39 smithi176 ceph-mon[36450]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-19T03:09:40.306 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:39 smithi176 ceph-mon[36450]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi059.front.sepia.ceph.com:3000"}]: dispatch 2024-03-19T03:09:40.306 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:39 smithi176 ceph-mon[36450]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-19T03:09:40.307 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:39 smithi176 ceph-mon[36450]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi059.front.sepia.ceph.com:9095"}]: dispatch 2024-03-19T03:09:40.307 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:39 smithi176 ceph-mon[36450]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:40.307 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:39 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:40.307 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:39 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:41.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:41 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:41.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:41 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:41.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:41 smithi176 ceph-mon[36450]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:41.793 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:41 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:41.794 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:41 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:41.794 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:41 smithi059 ceph-mon[29133]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:44.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:43 smithi059 ceph-mon[29133]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:44.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:43 smithi059 ceph-mon[29133]: from='osd.0 [v2:172.21.15.59:6802/1770897334,v1:172.21.15.59:6803/1770897334]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-19T03:09:44.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:44.168+0000 7faa48ff9700 1 -- 172.21.15.59:0/1864649593 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7faa4c0f8f40 con 0x7faa38069210 2024-03-19T03:09:44.169 INFO:teuthology.orchestra.run.smithi059.stdout:Created osd(s) 0 on host 'smithi059' 2024-03-19T03:09:44.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:44.169+0000 7faa523fc700 1 -- 172.21.15.59:0/1864649593 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7faa38069210 msgr2=0x7faa3806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:09:44.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:44.169+0000 7faa523fc700 1 --2- 172.21.15.59:0/1864649593 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7faa38069210 0x7faa3806b6c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7faa34007fd0 tx=0x7faa34009040 comp rx=0 tx=0).stop 2024-03-19T03:09:44.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:44.169+0000 7faa523fc700 1 -- 172.21.15.59:0/1864649593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa4c066f50 msgr2=0x7faa4c10fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:09:44.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:44.169+0000 7faa523fc700 1 --2- 172.21.15.59:0/1864649593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa4c066f50 0x7faa4c10fb80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00b970 tx=0x7faa3c009e30 comp rx=0 tx=0).stop 2024-03-19T03:09:44.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:44.169+0000 7faa523fc700 1 -- 172.21.15.59:0/1864649593 shutdown_connections 2024-03-19T03:09:44.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:44.169+0000 7faa523fc700 1 --2- 172.21.15.59:0/1864649593 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7faa38069210 0x7faa3806b6c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:09:44.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:44.169+0000 7faa523fc700 1 --2- 172.21.15.59:0/1864649593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa4c066f50 0x7faa4c10fb80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:09:44.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:44.169+0000 7faa523fc700 1 --2- 172.21.15.59:0/1864649593 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faa4c102300 0x7faa4c1100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:09:44.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:44.169+0000 7faa523fc700 1 -- 172.21.15.59:0/1864649593 >> 172.21.15.59:0/1864649593 conn(0x7faa4c0f7450 msgr2=0x7faa4c0f7820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:09:44.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:44.170+0000 7faa523fc700 1 -- 172.21.15.59:0/1864649593 shutdown_connections 2024-03-19T03:09:44.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:44.170+0000 7faa523fc700 1 -- 172.21.15.59:0/1864649593 wait complete. 2024-03-19T03:09:44.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:43 smithi176 ceph-mon[36450]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:44.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:43 smithi176 ceph-mon[36450]: from='osd.0 [v2:172.21.15.59:6802/1770897334,v1:172.21.15.59:6803/1770897334]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-19T03:09:44.929 DEBUG:teuthology.orchestra.run.smithi059:osd.0> sudo journalctl -f -n 0 -u ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@osd.0.service 2024-03-19T03:09:44.932 INFO:tasks.cephadm:Deploying osd.1 on smithi059 with /dev/vg_nvme/lv_3... 2024-03-19T03:09:44.932 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-19T03:09:45.215 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:44 smithi059 ceph-mon[29133]: from='osd.0 [v2:172.21.15.59:6802/1770897334,v1:172.21.15.59:6803/1770897334]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-19T03:09:45.215 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:44 smithi059 ceph-mon[29133]: osdmap e7: 1 total, 0 up, 1 in 2024-03-19T03:09:45.215 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:44 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:09:45.215 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:44 smithi059 ceph-mon[29133]: from='osd.0 [v2:172.21.15.59:6802/1770897334,v1:172.21.15.59:6803/1770897334]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-19T03:09:45.215 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:44 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:45.215 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:44 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:45.216 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:44 smithi059 ceph-mon[29133]: from='osd.0 [v2:172.21.15.59:6802/1770897334,v1:172.21.15.59:6803/1770897334]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-03-19T03:09:45.216 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:44 smithi059 ceph-mon[29133]: osdmap e8: 1 total, 0 up, 1 in 2024-03-19T03:09:45.216 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:44 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:09:45.216 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:09:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:09:44.978+0000 7f57fdaeb700 -1 osd.0 0 waiting for initial osdmap 2024-03-19T03:09:45.216 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:09:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:09:44.984+0000 7f57f80e5700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T03:09:45.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:44 smithi176 ceph-mon[36450]: from='osd.0 [v2:172.21.15.59:6802/1770897334,v1:172.21.15.59:6803/1770897334]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-19T03:09:45.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:44 smithi176 ceph-mon[36450]: osdmap e7: 1 total, 0 up, 1 in 2024-03-19T03:09:45.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:44 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:09:45.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:44 smithi176 ceph-mon[36450]: from='osd.0 [v2:172.21.15.59:6802/1770897334,v1:172.21.15.59:6803/1770897334]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-19T03:09:45.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:44 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:45.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:44 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:45.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:44 smithi176 ceph-mon[36450]: from='osd.0 [v2:172.21.15.59:6802/1770897334,v1:172.21.15.59:6803/1770897334]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-03-19T03:09:45.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:44 smithi176 ceph-mon[36450]: osdmap e8: 1 total, 0 up, 1 in 2024-03-19T03:09:45.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:44 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:09:45.611 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:09:46.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:45 smithi059 ceph-mon[29133]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:46.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:45 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:09:46.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:45 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:46.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:45 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:46.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:45 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:09:46.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:45 smithi176 ceph-mon[36450]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:46.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:45 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:09:46.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:45 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:46.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:45 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:46.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:45 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:09:47.287 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:46 smithi059 ceph-mon[29133]: purged_snaps scrub starts 2024-03-19T03:09:47.287 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:46 smithi059 ceph-mon[29133]: purged_snaps scrub ok 2024-03-19T03:09:47.287 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:09:47.287 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:46 smithi059 ceph-mon[29133]: osd.0 [v2:172.21.15.59:6802/1770897334,v1:172.21.15.59:6803/1770897334] boot 2024-03-19T03:09:47.287 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:46 smithi059 ceph-mon[29133]: osdmap e9: 1 total, 1 up, 1 in 2024-03-19T03:09:47.287 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:09:47.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:46 smithi176 ceph-mon[36450]: purged_snaps scrub starts 2024-03-19T03:09:47.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:46 smithi176 ceph-mon[36450]: purged_snaps scrub ok 2024-03-19T03:09:47.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:46 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:09:47.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:46 smithi176 ceph-mon[36450]: osd.0 [v2:172.21.15.59:6802/1770897334,v1:172.21.15.59:6803/1770897334] boot 2024-03-19T03:09:47.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:46 smithi176 ceph-mon[36450]: osdmap e9: 1 total, 1 up, 1 in 2024-03-19T03:09:47.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:46 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:09:47.585 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-19T03:09:47.585 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-19T03:09:47.585 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10+0 records in 2024-03-19T03:09:47.585 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2024-03-19T03:09:47.585 INFO:teuthology.orchestra.run.smithi059.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123007 s, 852 MB/s 2024-03-19T03:09:47.586 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping successful for: 2024-03-19T03:09:48.238 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:47 smithi059 ceph-mon[29133]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:48.238 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:47 smithi059 ceph-mon[29133]: osdmap e10: 1 total, 1 up, 1 in 2024-03-19T03:09:48.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:47 smithi176 ceph-mon[36450]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:09:48.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:47 smithi176 ceph-mon[36450]: osdmap e10: 1 total, 1 up, 1 in 2024-03-19T03:09:49.332 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi059:vg_nvme/lv_3 2024-03-19T03:09:50.119 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:49 smithi059 ceph-mon[29133]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:09:50.211 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:09:50.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:49 smithi176 ceph-mon[36450]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:09:51.253 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:51 smithi059 ceph-mon[29133]: Detected new or changed devices on smithi059 2024-03-19T03:09:51.253 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:51 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:51.253 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:51 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:51.253 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:51 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:09:51.253 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:51 smithi059 ceph-mon[29133]: Adjusting osd_memory_target on smithi059 to 12869M 2024-03-19T03:09:51.253 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:51 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:51.253 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:51 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:51.253 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:51 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:09:51.253 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:51 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:51.253 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:51 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:09:51.535 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:51 smithi176 ceph-mon[36450]: Detected new or changed devices on smithi059 2024-03-19T03:09:51.535 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:51 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:51.535 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:51 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:51.535 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:51 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:09:51.535 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:51 smithi176 ceph-mon[36450]: Adjusting osd_memory_target on smithi059 to 12869M 2024-03-19T03:09:51.536 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:51 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:51.536 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:51 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:51.536 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:51 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:09:51.536 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:51 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:51.536 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:51 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:09:51.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.735+0000 7f1c6e0ba700 1 -- 172.21.15.59:0/2981505187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c680fdff0 msgr2=0x7f1c680fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:09:51.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.735+0000 7f1c6e0ba700 1 --2- 172.21.15.59:0/2981505187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c680fdff0 0x7f1c680fe460 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c004940 tx=0x7f1c5c009e30 comp rx=0 tx=0).stop 2024-03-19T03:09:51.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.735+0000 7f1c6e0ba700 1 -- 172.21.15.59:0/2981505187 shutdown_connections 2024-03-19T03:09:51.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.735+0000 7f1c6e0ba700 1 --2- 172.21.15.59:0/2981505187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c680fdff0 0x7f1c680fe460 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:09:51.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.735+0000 7f1c6e0ba700 1 --2- 172.21.15.59:0/2981505187 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c680fce80 0x7f1c680fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:09:51.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.735+0000 7f1c6e0ba700 1 -- 172.21.15.59:0/2981505187 >> 172.21.15.59:0/2981505187 conn(0x7f1c680f8580 msgr2=0x7f1c680fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:09:51.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.735+0000 7f1c6e0ba700 1 -- 172.21.15.59:0/2981505187 shutdown_connections 2024-03-19T03:09:51.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.735+0000 7f1c6e0ba700 1 -- 172.21.15.59:0/2981505187 wait complete. 2024-03-19T03:09:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.736+0000 7f1c6e0ba700 1 Processor -- start 2024-03-19T03:09:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.736+0000 7f1c6e0ba700 1 -- start start 2024-03-19T03:09:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.736+0000 7f1c6e0ba700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c680fce80 0x7f1c6810da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:09:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.736+0000 7f1c6e0ba700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c680fdff0 0x7f1c6810df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:09:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.736+0000 7f1c6e0ba700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6810e550 con 0x7f1c680fce80 2024-03-19T03:09:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.736+0000 7f1c6e0ba700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6810eda0 con 0x7f1c680fdff0 2024-03-19T03:09:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.737+0000 7f1c677fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c680fce80 0x7f1c6810da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:09:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.737+0000 7f1c677fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c680fce80 0x7f1c6810da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50496/0 (socket says 172.21.15.59:50496) 2024-03-19T03:09:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.737+0000 7f1c677fe700 1 -- 172.21.15.59:0/247428687 learned_addr learned my addr 172.21.15.59:0/247428687 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:09:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.737+0000 7f1c66ffd700 1 --2- 172.21.15.59:0/247428687 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c680fdff0 0x7f1c6810df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:09:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.737+0000 7f1c677fe700 1 -- 172.21.15.59:0/247428687 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c680fdff0 msgr2=0x7f1c6810df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:09:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.737+0000 7f1c677fe700 1 --2- 172.21.15.59:0/247428687 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c680fdff0 0x7f1c6810df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:09:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.737+0000 7f1c677fe700 1 -- 172.21.15.59:0/247428687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5c005040 con 0x7f1c680fce80 2024-03-19T03:09:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.737+0000 7f1c677fe700 1 --2- 172.21.15.59:0/247428687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c680fce80 0x7f1c6810da10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1c58002a50 tx=0x7f1c580046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:09:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.737+0000 7f1c4ffff700 1 -- 172.21.15.59:0/247428687 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c58007cc0 con 0x7f1c680fce80 2024-03-19T03:09:51.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.737+0000 7f1c6e0ba700 1 -- 172.21.15.59:0/247428687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c6810f080 con 0x7f1c680fce80 2024-03-19T03:09:51.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.737+0000 7f1c4ffff700 1 -- 172.21.15.59:0/247428687 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1c5800f040 con 0x7f1c680fce80 2024-03-19T03:09:51.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.737+0000 7f1c4ffff700 1 -- 172.21.15.59:0/247428687 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c5800da30 con 0x7f1c680fce80 2024-03-19T03:09:51.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.737+0000 7f1c6e0ba700 1 -- 172.21.15.59:0/247428687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c6810f5c0 con 0x7f1c680fce80 2024-03-19T03:09:51.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.738+0000 7f1c6e0ba700 1 -- 172.21.15.59:0/247428687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c6805f5f0 con 0x7f1c680fce80 2024-03-19T03:09:51.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.739+0000 7f1c4ffff700 1 -- 172.21.15.59:0/247428687 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f1c5800db90 con 0x7f1c680fce80 2024-03-19T03:09:51.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.740+0000 7f1c4ffff700 1 --2- 172.21.15.59:0/247428687 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f1c50069210 0x7f1c5006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:09:51.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.740+0000 7f1c4ffff700 1 -- 172.21.15.59:0/247428687 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7f1c58085fb0 con 0x7f1c680fce80 2024-03-19T03:09:51.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.740+0000 7f1c66ffd700 1 --2- 172.21.15.59:0/247428687 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f1c50069210 0x7f1c5006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:09:51.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.740+0000 7f1c66ffd700 1 --2- 172.21.15.59:0/247428687 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f1c50069210 0x7f1c5006b6c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c009fd0 tx=0x7f1c5c009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:09:51.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.744+0000 7f1c4ffff700 1 -- 172.21.15.59:0/247428687 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f1c58056bf0 con 0x7f1c680fce80 2024-03-19T03:09:51.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:09:51.950+0000 7f1c6e0ba700 1 -- 172.21.15.59:0/247428687 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c6810f990 con 0x7f1c50069210 2024-03-19T03:09:52.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:52 smithi059 ceph-mon[29133]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:09:52.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:52 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:09:52.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:52 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:09:52.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:52 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:52.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:52 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:52.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:52 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:52.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:52 smithi176 ceph-mon[36450]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:09:52.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:52 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:09:52.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:52 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:09:52.629 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:52 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:52.629 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:52 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:52.629 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:52 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:53.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:53 smithi059 ceph-mon[29133]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:09:53.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:53 smithi176 ceph-mon[36450]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:09:54.498 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:54 smithi059 ceph-mon[29133]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:09:54.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:54 smithi176 ceph-mon[36450]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:09:56.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:56 smithi176 ceph-mon[36450]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:09:56.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:56 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/2423621655' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1069548-eb23-476b-a661-b71f0a73987e"}]: dispatch 2024-03-19T03:09:56.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:56 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/2423621655' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1069548-eb23-476b-a661-b71f0a73987e"}]': finished 2024-03-19T03:09:56.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:56 smithi176 ceph-mon[36450]: osdmap e11: 2 total, 1 up, 2 in 2024-03-19T03:09:56.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:56 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:09:56.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:56 smithi059 ceph-mon[29133]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:09:56.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:56 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/2423621655' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1069548-eb23-476b-a661-b71f0a73987e"}]: dispatch 2024-03-19T03:09:56.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:56 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/2423621655' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1069548-eb23-476b-a661-b71f0a73987e"}]': finished 2024-03-19T03:09:56.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:56 smithi059 ceph-mon[29133]: osdmap e11: 2 total, 1 up, 2 in 2024-03-19T03:09:56.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:56 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:09:57.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:57 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/3527744393' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:09:57.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:57 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:57.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:57 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:57.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:57 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:57.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:57 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:09:57.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:57 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:57.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:57 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/3527744393' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:09:57.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:57 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:57.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:57 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:57.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:57 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:09:57.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:57 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:09:57.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:57 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:09:58.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:09:58 smithi059 ceph-mon[29133]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:09:58.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:09:58 smithi176 ceph-mon[36450]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:10:00.549 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:00 smithi059 ceph-mon[29133]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:10:00.549 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:00 smithi059 ceph-mon[29133]: overall HEALTH_OK 2024-03-19T03:10:00.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:00 smithi176 ceph-mon[36450]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:10:00.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:00 smithi176 ceph-mon[36450]: overall HEALTH_OK 2024-03-19T03:10:01.528 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:10:01.615 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:10:02.578 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:02 smithi059 ceph-mon[29133]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:10:02.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:02 smithi176 ceph-mon[36450]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:10:03.521 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:03 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-19T03:10:03.522 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:03 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:10:03.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:03 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-19T03:10:03.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:03 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:10:04.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:04 smithi059 ceph-mon[29133]: Deploying daemon osd.1 on smithi059 2024-03-19T03:10:04.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:04 smithi059 ceph-mon[29133]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:10:04.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:04 smithi176 ceph-mon[36450]: Deploying daemon osd.1 on smithi059 2024-03-19T03:10:04.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:04 smithi176 ceph-mon[36450]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:10:06.275 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:06 smithi059 ceph-mon[29133]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:10:06.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:06 smithi176 ceph-mon[36450]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:10:08.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:08 smithi059 ceph-mon[29133]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:10:08.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:08 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:08.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:08 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:08.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:08 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:10:08.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:08 smithi176 ceph-mon[36450]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:10:08.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:08 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:08.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:08 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:08.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:08 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:10:10.388 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:10 smithi059 ceph-mon[29133]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:10:10.388 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:10 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:10.388 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:10 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:10.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:10 smithi176 ceph-mon[36450]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:10:10.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:10 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:10.629 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:10 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:11.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:11.403+0000 7f1c4ffff700 1 -- 172.21.15.59:0/247428687 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1c6810f990 con 0x7f1c50069210 2024-03-19T03:10:11.405 INFO:teuthology.orchestra.run.smithi059.stdout:Created osd(s) 1 on host 'smithi059' 2024-03-19T03:10:11.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:11.405+0000 7f1c6e0ba700 1 -- 172.21.15.59:0/247428687 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f1c50069210 msgr2=0x7f1c5006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:10:11.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:11.405+0000 7f1c6e0ba700 1 --2- 172.21.15.59:0/247428687 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f1c50069210 0x7f1c5006b6c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c009fd0 tx=0x7f1c5c009e30 comp rx=0 tx=0).stop 2024-03-19T03:10:11.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:11.405+0000 7f1c6e0ba700 1 -- 172.21.15.59:0/247428687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c680fce80 msgr2=0x7f1c6810da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:10:11.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:11.405+0000 7f1c6e0ba700 1 --2- 172.21.15.59:0/247428687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c680fce80 0x7f1c6810da10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1c58002a50 tx=0x7f1c580046b0 comp rx=0 tx=0).stop 2024-03-19T03:10:11.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:11.405+0000 7f1c6e0ba700 1 -- 172.21.15.59:0/247428687 shutdown_connections 2024-03-19T03:10:11.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:11.405+0000 7f1c6e0ba700 1 --2- 172.21.15.59:0/247428687 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f1c50069210 0x7f1c5006b6c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:10:11.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:11.405+0000 7f1c6e0ba700 1 --2- 172.21.15.59:0/247428687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c680fce80 0x7f1c6810da10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:10:11.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:11.405+0000 7f1c6e0ba700 1 --2- 172.21.15.59:0/247428687 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c680fdff0 0x7f1c6810df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:10:11.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:11.405+0000 7f1c6e0ba700 1 -- 172.21.15.59:0/247428687 >> 172.21.15.59:0/247428687 conn(0x7f1c680f8580 msgr2=0x7f1c680f9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:10:11.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:11.405+0000 7f1c6e0ba700 1 -- 172.21.15.59:0/247428687 shutdown_connections 2024-03-19T03:10:11.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:11.405+0000 7f1c6e0ba700 1 -- 172.21.15.59:0/247428687 wait complete. 2024-03-19T03:10:12.401 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:12 smithi059 ceph-mon[29133]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:10:12.401 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:12 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:12.401 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:12 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:12.401 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:12 smithi059 ceph-mon[29133]: from='osd.1 [v2:172.21.15.59:6810/3595398218,v1:172.21.15.59:6811/3595398218]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T03:10:12.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:12 smithi176 ceph-mon[36450]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:10:12.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:12 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:12.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:12 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:12.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:12 smithi176 ceph-mon[36450]: from='osd.1 [v2:172.21.15.59:6810/3595398218,v1:172.21.15.59:6811/3595398218]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T03:10:12.947 DEBUG:teuthology.orchestra.run.smithi059:osd.1> sudo journalctl -f -n 0 -u ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@osd.1.service 2024-03-19T03:10:12.950 INFO:tasks.cephadm:Deploying osd.2 on smithi059 with /dev/vg_nvme/lv_2... 2024-03-19T03:10:12.950 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-19T03:10:13.333 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:10:13.457 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:13 smithi059 ceph-mon[29133]: from='osd.1 [v2:172.21.15.59:6810/3595398218,v1:172.21.15.59:6811/3595398218]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-19T03:10:13.457 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:13 smithi059 ceph-mon[29133]: osdmap e12: 2 total, 1 up, 2 in 2024-03-19T03:10:13.457 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:13 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:10:13.457 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:13 smithi059 ceph-mon[29133]: from='osd.1 [v2:172.21.15.59:6810/3595398218,v1:172.21.15.59:6811/3595398218]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-19T03:10:13.457 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:13 smithi059 ceph-mon[29133]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:10:13.457 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:10:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:10:13.394+0000 7fc4a17f5700 -1 osd.1 0 waiting for initial osdmap 2024-03-19T03:10:13.457 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:10:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:10:13.400+0000 7fc49aded700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T03:10:13.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:13 smithi176 ceph-mon[36450]: from='osd.1 [v2:172.21.15.59:6810/3595398218,v1:172.21.15.59:6811/3595398218]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-19T03:10:13.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:13 smithi176 ceph-mon[36450]: osdmap e12: 2 total, 1 up, 2 in 2024-03-19T03:10:13.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:13 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:10:13.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:13 smithi176 ceph-mon[36450]: from='osd.1 [v2:172.21.15.59:6810/3595398218,v1:172.21.15.59:6811/3595398218]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-19T03:10:13.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:13 smithi176 ceph-mon[36450]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:10:14.759 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:14 smithi059 ceph-mon[29133]: from='osd.1 [v2:172.21.15.59:6810/3595398218,v1:172.21.15.59:6811/3595398218]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-03-19T03:10:14.760 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:14 smithi059 ceph-mon[29133]: osdmap e13: 2 total, 1 up, 2 in 2024-03-19T03:10:14.760 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:14 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:10:14.760 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:14 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:10:14.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:14 smithi176 ceph-mon[36450]: from='osd.1 [v2:172.21.15.59:6810/3595398218,v1:172.21.15.59:6811/3595398218]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-03-19T03:10:14.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:14 smithi176 ceph-mon[36450]: osdmap e13: 2 total, 1 up, 2 in 2024-03-19T03:10:14.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:14 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:10:14.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:14 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:10:15.460 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-19T03:10:15.460 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-19T03:10:15.460 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10+0 records in 2024-03-19T03:10:15.460 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2024-03-19T03:10:15.460 INFO:teuthology.orchestra.run.smithi059.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126035 s, 832 MB/s 2024-03-19T03:10:15.460 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping successful for: 2024-03-19T03:10:15.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:15 smithi059 ceph-mon[29133]: purged_snaps scrub starts 2024-03-19T03:10:15.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:15 smithi059 ceph-mon[29133]: purged_snaps scrub ok 2024-03-19T03:10:15.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:15 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:10:15.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:15 smithi059 ceph-mon[29133]: osd.1 [v2:172.21.15.59:6810/3595398218,v1:172.21.15.59:6811/3595398218] boot 2024-03-19T03:10:15.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:15 smithi059 ceph-mon[29133]: osdmap e14: 2 total, 2 up, 2 in 2024-03-19T03:10:15.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:15 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:10:15.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:15 smithi059 ceph-mon[29133]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:10:15.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:15 smithi176 ceph-mon[36450]: purged_snaps scrub starts 2024-03-19T03:10:15.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:15 smithi176 ceph-mon[36450]: purged_snaps scrub ok 2024-03-19T03:10:15.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:15 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:10:15.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:15 smithi176 ceph-mon[36450]: osd.1 [v2:172.21.15.59:6810/3595398218,v1:172.21.15.59:6811/3595398218] boot 2024-03-19T03:10:15.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:15 smithi176 ceph-mon[36450]: osdmap e14: 2 total, 2 up, 2 in 2024-03-19T03:10:15.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:15 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:10:15.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:15 smithi176 ceph-mon[36450]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:10:16.398 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi059:vg_nvme/lv_2 2024-03-19T03:10:16.632 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:16 smithi059 ceph-mon[29133]: osdmap e15: 2 total, 2 up, 2 in 2024-03-19T03:10:16.632 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:16 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:16.632 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:16 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:16.632 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:16 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:10:16.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:16 smithi176 ceph-mon[36450]: osdmap e15: 2 total, 2 up, 2 in 2024-03-19T03:10:16.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:16 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:16.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:16 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:16.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:16 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:10:17.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:17 smithi059 ceph-mon[29133]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:17.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:17 smithi176 ceph-mon[36450]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:18.218 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:10:20.163 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:19 smithi059 ceph-mon[29133]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:20.163 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:19 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:20.164 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:19 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:20.164 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:19 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:10:20.164 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:19 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:10:20.164 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:19 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:20.164 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:19 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:10:20.164 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:19 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:10:20.164 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:19 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:20.164 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:19 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:10:20.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:19 smithi176 ceph-mon[36450]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:20.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:19 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:20.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:19 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:20.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:19 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:10:20.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:19 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:10:20.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:19 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:20.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:19 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:10:20.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:19 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:10:20.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:19 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:20.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:19 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:10:20.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.380+0000 7fe3e24eb700 1 -- 172.21.15.59:0/1694778204 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc0fcf80 msgr2=0x7fe3dc0fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:10:20.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.380+0000 7fe3e24eb700 1 --2- 172.21.15.59:0/1694778204 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc0fcf80 0x7fe3dc0fd3f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0004940 tx=0x7fe3d0009e30 comp rx=0 tx=0).stop 2024-03-19T03:10:20.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.380+0000 7fe3e24eb700 1 -- 172.21.15.59:0/1694778204 shutdown_connections 2024-03-19T03:10:20.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.380+0000 7fe3e24eb700 1 --2- 172.21.15.59:0/1694778204 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc0fcf80 0x7fe3dc0fd3f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:10:20.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.380+0000 7fe3e24eb700 1 --2- 172.21.15.59:0/1694778204 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3dc0fbcd0 0x7fe3dc0fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:10:20.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.380+0000 7fe3e24eb700 1 -- 172.21.15.59:0/1694778204 >> 172.21.15.59:0/1694778204 conn(0x7fe3dc0f7430 msgr2=0x7fe3dc0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:10:20.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.380+0000 7fe3e24eb700 1 -- 172.21.15.59:0/1694778204 shutdown_connections 2024-03-19T03:10:20.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.380+0000 7fe3e24eb700 1 -- 172.21.15.59:0/1694778204 wait complete. 2024-03-19T03:10:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.381+0000 7fe3e24eb700 1 Processor -- start 2024-03-19T03:10:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.381+0000 7fe3e24eb700 1 -- start start 2024-03-19T03:10:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.381+0000 7fe3e24eb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc0fbcd0 0x7fe3dc066cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:10:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.381+0000 7fe3e24eb700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3dc0fcf80 0x7fe3dc112720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:10:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.381+0000 7fe3e24eb700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc112c60 con 0x7fe3dc0fbcd0 2024-03-19T03:10:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.381+0000 7fe3e24eb700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc112dd0 con 0x7fe3dc0fcf80 2024-03-19T03:10:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.381+0000 7fe3dbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc0fbcd0 0x7fe3dc066cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:10:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.382+0000 7fe3dbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc0fbcd0 0x7fe3dc066cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48260/0 (socket says 172.21.15.59:48260) 2024-03-19T03:10:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.382+0000 7fe3dbfff700 1 -- 172.21.15.59:0/1177808714 learned_addr learned my addr 172.21.15.59:0/1177808714 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:10:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.382+0000 7fe3db7fe700 1 --2- 172.21.15.59:0/1177808714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3dc0fcf80 0x7fe3dc112720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:10:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.382+0000 7fe3dbfff700 1 -- 172.21.15.59:0/1177808714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3dc0fcf80 msgr2=0x7fe3dc112720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:10:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.382+0000 7fe3dbfff700 1 --2- 172.21.15.59:0/1177808714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3dc0fcf80 0x7fe3dc112720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:10:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.382+0000 7fe3dbfff700 1 -- 172.21.15.59:0/1177808714 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d0005040 con 0x7fe3dc0fbcd0 2024-03-19T03:10:20.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.382+0000 7fe3dbfff700 1 --2- 172.21.15.59:0/1177808714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc0fbcd0 0x7fe3dc066cf0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc00b790 tx=0x7fe3cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:10:20.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.382+0000 7fe3d8ff9700 1 -- 172.21.15.59:0/1177808714 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3cc00f6f0 con 0x7fe3dc0fbcd0 2024-03-19T03:10:20.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.382+0000 7fe3d8ff9700 1 -- 172.21.15.59:0/1177808714 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe3cc00fd20 con 0x7fe3dc0fbcd0 2024-03-19T03:10:20.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.382+0000 7fe3e24eb700 1 -- 172.21.15.59:0/1177808714 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3dc1130b0 con 0x7fe3dc0fbcd0 2024-03-19T03:10:20.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.382+0000 7fe3d8ff9700 1 -- 172.21.15.59:0/1177808714 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3cc0103a0 con 0x7fe3dc0fbcd0 2024-03-19T03:10:20.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.382+0000 7fe3e24eb700 1 -- 172.21.15.59:0/1177808714 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3dc113670 con 0x7fe3dc0fbcd0 2024-03-19T03:10:20.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.383+0000 7fe3e24eb700 1 -- 172.21.15.59:0/1177808714 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3dc04c710 con 0x7fe3dc0fbcd0 2024-03-19T03:10:20.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.383+0000 7fe3d8ff9700 1 -- 172.21.15.59:0/1177808714 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7fe3cc011400 con 0x7fe3dc0fbcd0 2024-03-19T03:10:20.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.384+0000 7fe3d8ff9700 1 --2- 172.21.15.59:0/1177808714 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe3c4069210 0x7fe3c406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:10:20.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.384+0000 7fe3d8ff9700 1 -- 172.21.15.59:0/1177808714 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7fe3cc086270 con 0x7fe3dc0fbcd0 2024-03-19T03:10:20.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.384+0000 7fe3db7fe700 1 --2- 172.21.15.59:0/1177808714 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe3c4069210 0x7fe3c406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:10:20.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.384+0000 7fe3db7fe700 1 --2- 172.21.15.59:0/1177808714 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe3c4069210 0x7fe3c406b6c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe3d000b970 tx=0x7fe3d0009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:10:20.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.388+0000 7fe3d8ff9700 1 -- 172.21.15.59:0/1177808714 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fe3cc056d00 con 0x7fe3dc0fbcd0 2024-03-19T03:10:20.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:20.592+0000 7fe3e24eb700 1 -- 172.21.15.59:0/1177808714 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3dc101250 con 0x7fe3c4069210 2024-03-19T03:10:21.444 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:21 smithi059 ceph-mon[29133]: Detected new or changed devices on smithi059 2024-03-19T03:10:21.445 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:21 smithi059 ceph-mon[29133]: Adjusting osd_memory_target on smithi059 to 6434M 2024-03-19T03:10:21.445 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:21 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:10:21.445 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:21 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:10:21.445 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:21 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:10:21.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:21 smithi176 ceph-mon[36450]: Detected new or changed devices on smithi059 2024-03-19T03:10:21.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:21 smithi176 ceph-mon[36450]: Adjusting osd_memory_target on smithi059 to 6434M 2024-03-19T03:10:21.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:21 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:10:21.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:21 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:10:21.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:21 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:10:22.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:22 smithi059 ceph-mon[29133]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:10:22.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:22 smithi059 ceph-mon[29133]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:22.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:22 smithi176 ceph-mon[36450]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:10:22.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:22 smithi176 ceph-mon[36450]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:24.184 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:24 smithi059 ceph-mon[29133]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:24.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:24 smithi176 ceph-mon[36450]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:25.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:25 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/3991558542' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4958f1e4-2dc9-458f-82ff-e51d98762cd2"}]: dispatch 2024-03-19T03:10:25.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:25 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/3991558542' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4958f1e4-2dc9-458f-82ff-e51d98762cd2"}]': finished 2024-03-19T03:10:25.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:25 smithi059 ceph-mon[29133]: osdmap e16: 3 total, 2 up, 3 in 2024-03-19T03:10:25.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:25 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:10:25.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:25 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/3991558542' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4958f1e4-2dc9-458f-82ff-e51d98762cd2"}]: dispatch 2024-03-19T03:10:25.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:25 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/3991558542' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4958f1e4-2dc9-458f-82ff-e51d98762cd2"}]': finished 2024-03-19T03:10:25.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:25 smithi176 ceph-mon[36450]: osdmap e16: 3 total, 2 up, 3 in 2024-03-19T03:10:25.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:25 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:10:26.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:26 smithi176 ceph-mon[36450]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:26.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:26 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/1017117571' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:10:26.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:26 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:26.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:26 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:26.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:26 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:10:26.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:26 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:10:26.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:26 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:26.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:26 smithi059 ceph-mon[29133]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:26.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:26 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/1017117571' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:10:26.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:26 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:26.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:26 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:26.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:26 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:10:26.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:26 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:10:26.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:26 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:28.491 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:28 smithi059 ceph-mon[29133]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:28.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:28 smithi176 ceph-mon[36450]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:30.290 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:30 smithi059 ceph-mon[29133]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:30.430 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:30 smithi176 ceph-mon[36450]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:31.309 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:10:31.615 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:31 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:10:32.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:32 smithi059 ceph-mon[29133]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:32.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:32 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-19T03:10:32.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:32 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:10:32.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:32 smithi176 ceph-mon[36450]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:32.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:32 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-19T03:10:32.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:32 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:10:33.410 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:33 smithi059 ceph-mon[29133]: Deploying daemon osd.2 on smithi059 2024-03-19T03:10:33.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:33 smithi176 ceph-mon[36450]: Deploying daemon osd.2 on smithi059 2024-03-19T03:10:34.410 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:34 smithi059 ceph-mon[29133]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:34.630 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:34 smithi176 ceph-mon[36450]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:36.284 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:36 smithi059 ceph-mon[29133]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:36.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:36 smithi176 ceph-mon[36450]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:38.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:38 smithi059 ceph-mon[29133]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:38.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:38.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:38.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:10:38.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:38 smithi176 ceph-mon[36450]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:38.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:38.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:38.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:10:40.153 INFO:teuthology.orchestra.run.smithi059.stdout:Created osd(s) 2 on host 'smithi059' 2024-03-19T03:10:40.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:40.150+0000 7fe3d8ff9700 1 -- 172.21.15.59:0/1177808714 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe3dc101250 con 0x7fe3c4069210 2024-03-19T03:10:40.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:40.152+0000 7fe3e24eb700 1 -- 172.21.15.59:0/1177808714 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe3c4069210 msgr2=0x7fe3c406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:10:40.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:40.152+0000 7fe3e24eb700 1 --2- 172.21.15.59:0/1177808714 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe3c4069210 0x7fe3c406b6c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe3d000b970 tx=0x7fe3d0009e30 comp rx=0 tx=0).stop 2024-03-19T03:10:40.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:40.152+0000 7fe3e24eb700 1 -- 172.21.15.59:0/1177808714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc0fbcd0 msgr2=0x7fe3dc066cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:10:40.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:40.152+0000 7fe3e24eb700 1 --2- 172.21.15.59:0/1177808714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc0fbcd0 0x7fe3dc066cf0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc00b790 tx=0x7fe3cc009250 comp rx=0 tx=0).stop 2024-03-19T03:10:40.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:40.152+0000 7fe3e24eb700 1 -- 172.21.15.59:0/1177808714 shutdown_connections 2024-03-19T03:10:40.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:40.152+0000 7fe3e24eb700 1 --2- 172.21.15.59:0/1177808714 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe3c4069210 0x7fe3c406b6c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:10:40.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:40.152+0000 7fe3e24eb700 1 --2- 172.21.15.59:0/1177808714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc0fbcd0 0x7fe3dc066cf0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:10:40.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:40.152+0000 7fe3e24eb700 1 --2- 172.21.15.59:0/1177808714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3dc0fcf80 0x7fe3dc112720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:10:40.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:40.152+0000 7fe3e24eb700 1 -- 172.21.15.59:0/1177808714 >> 172.21.15.59:0/1177808714 conn(0x7fe3dc0f7430 msgr2=0x7fe3dc0f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:10:40.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:40.153+0000 7fe3e24eb700 1 -- 172.21.15.59:0/1177808714 shutdown_connections 2024-03-19T03:10:40.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:10:40.153+0000 7fe3e24eb700 1 -- 172.21.15.59:0/1177808714 wait complete. 2024-03-19T03:10:40.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:40 smithi176 ceph-mon[36450]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:40.507 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:40 smithi059 ceph-mon[29133]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:40.841 DEBUG:teuthology.orchestra.run.smithi059:osd.2> sudo journalctl -f -n 0 -u ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@osd.2.service 2024-03-19T03:10:40.844 INFO:tasks.cephadm:Deploying osd.3 on smithi176 with /dev/vg_nvme/lv_4... 2024-03-19T03:10:40.844 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-19T03:10:41.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:10:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:10:41.003+0000 7fa7ea604540 -1 osd.2 0 log_to_monitors true 2024-03-19T03:10:41.105 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi176/config 2024-03-19T03:10:41.425 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:41 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:41.425 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:41 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:41.425 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:41 smithi176 ceph-mon[36450]: from='osd.2 [v2:172.21.15.59:6818/639205676,v1:172.21.15.59:6819/639205676]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T03:10:41.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:41 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:41.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:41 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:41.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:41 smithi059 ceph-mon[29133]: from='osd.2 [v2:172.21.15.59:6818/639205676,v1:172.21.15.59:6819/639205676]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T03:10:42.426 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:42 smithi176 ceph-mon[36450]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:42.426 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:42 smithi176 ceph-mon[36450]: from='osd.2 [v2:172.21.15.59:6818/639205676,v1:172.21.15.59:6819/639205676]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-19T03:10:42.426 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:42 smithi176 ceph-mon[36450]: osdmap e17: 3 total, 2 up, 3 in 2024-03-19T03:10:42.427 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:42 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:10:42.427 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:42 smithi176 ceph-mon[36450]: from='osd.2 [v2:172.21.15.59:6818/639205676,v1:172.21.15.59:6819/639205676]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-19T03:10:42.516 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:42 smithi059 ceph-mon[29133]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:42.516 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:42 smithi059 ceph-mon[29133]: from='osd.2 [v2:172.21.15.59:6818/639205676,v1:172.21.15.59:6819/639205676]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-19T03:10:42.516 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:42 smithi059 ceph-mon[29133]: osdmap e17: 3 total, 2 up, 3 in 2024-03-19T03:10:42.516 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:42 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:10:42.516 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:42 smithi059 ceph-mon[29133]: from='osd.2 [v2:172.21.15.59:6818/639205676,v1:172.21.15.59:6819/639205676]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-03-19T03:10:42.517 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:10:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:10:42.154+0000 7fa7e0dd6700 -1 osd.2 0 waiting for initial osdmap 2024-03-19T03:10:42.517 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:10:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:10:42.158+0000 7fa7d9bcd700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T03:10:42.959 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-19T03:10:42.959 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-19T03:10:42.959 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2024-03-19T03:10:42.959 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2024-03-19T03:10:42.959 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011438 s, 917 MB/s 2024-03-19T03:10:42.959 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2024-03-19T03:10:43.501 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:43 smithi176 ceph-mon[36450]: from='osd.2 [v2:172.21.15.59:6818/639205676,v1:172.21.15.59:6819/639205676]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-03-19T03:10:43.501 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:43 smithi176 ceph-mon[36450]: osdmap e18: 3 total, 2 up, 3 in 2024-03-19T03:10:43.501 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:43 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:10:43.501 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:43 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:10:43.525 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi176:vg_nvme/lv_4 2024-03-19T03:10:43.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:43 smithi059 ceph-mon[29133]: from='osd.2 [v2:172.21.15.59:6818/639205676,v1:172.21.15.59:6819/639205676]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-03-19T03:10:43.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:43 smithi059 ceph-mon[29133]: osdmap e18: 3 total, 2 up, 3 in 2024-03-19T03:10:43.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:43 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:10:43.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:43 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:10:43.782 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi176/config 2024-03-19T03:10:44.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:44 smithi176 ceph-mon[36450]: purged_snaps scrub starts 2024-03-19T03:10:44.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:44 smithi176 ceph-mon[36450]: purged_snaps scrub ok 2024-03-19T03:10:44.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:44 smithi176 ceph-mon[36450]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:44.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:44 smithi176 ceph-mon[36450]: osd.2 [v2:172.21.15.59:6818/639205676,v1:172.21.15.59:6819/639205676] boot 2024-03-19T03:10:44.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:44 smithi176 ceph-mon[36450]: osdmap e19: 3 total, 3 up, 3 in 2024-03-19T03:10:44.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:44 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:10:44.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:44 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:44.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:44 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:44.457 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:44 smithi059 ceph-mon[29133]: purged_snaps scrub starts 2024-03-19T03:10:44.457 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:44 smithi059 ceph-mon[29133]: purged_snaps scrub ok 2024-03-19T03:10:44.457 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:44 smithi059 ceph-mon[29133]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-19T03:10:44.457 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:44 smithi059 ceph-mon[29133]: osd.2 [v2:172.21.15.59:6818/639205676,v1:172.21.15.59:6819/639205676] boot 2024-03-19T03:10:44.457 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:44 smithi059 ceph-mon[29133]: osdmap e19: 3 total, 3 up, 3 in 2024-03-19T03:10:44.457 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:44 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:10:44.457 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:44 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:44.458 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:44 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:45.302 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.299+0000 7fd9602a6700 1 -- 172.21.15.176:0/1359745894 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9580662e0 msgr2=0x7fd9580ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:10:45.302 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.299+0000 7fd9602a6700 1 --2- 172.21.15.176:0/1359745894 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9580662e0 0x7fd9580ff9c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd954004940 tx=0x7fd954009e30 comp rx=0 tx=0).stop 2024-03-19T03:10:45.302 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.299+0000 7fd9602a6700 1 -- 172.21.15.176:0/1359745894 shutdown_connections 2024-03-19T03:10:45.302 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.299+0000 7fd9602a6700 1 --2- 172.21.15.176:0/1359745894 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9580fff00 0x7fd958102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:10:45.302 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.299+0000 7fd9602a6700 1 --2- 172.21.15.176:0/1359745894 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9580662e0 0x7fd9580ff9c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:10:45.303 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.299+0000 7fd9602a6700 1 -- 172.21.15.176:0/1359745894 >> 172.21.15.176:0/1359745894 conn(0x7fd9580f7430 msgr2=0x7fd9580f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:10:45.303 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.299+0000 7fd9602a6700 1 -- 172.21.15.176:0/1359745894 shutdown_connections 2024-03-19T03:10:45.303 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.299+0000 7fd9602a6700 1 -- 172.21.15.176:0/1359745894 wait complete. 2024-03-19T03:10:45.303 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.299+0000 7fd9602a6700 1 Processor -- start 2024-03-19T03:10:45.303 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.300+0000 7fd9602a6700 1 -- start start 2024-03-19T03:10:45.303 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.300+0000 7fd9602a6700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9580662e0 0x7fd9580fd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:10:45.303 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.300+0000 7fd9602a6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9580fff00 0x7fd9580fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:10:45.303 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.300+0000 7fd9602a6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9580fbe70 con 0x7fd9580fff00 2024-03-19T03:10:45.303 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.300+0000 7fd9602a6700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9580fbfe0 con 0x7fd9580662e0 2024-03-19T03:10:45.303 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.300+0000 7fd95e042700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9580662e0 0x7fd9580fd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:10:45.303 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.300+0000 7fd95e042700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9580662e0 0x7fd9580fd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:56868/0 (socket says 172.21.15.176:56868) 2024-03-19T03:10:45.303 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.300+0000 7fd95e042700 1 -- 172.21.15.176:0/2579660255 learned_addr learned my addr 172.21.15.176:0/2579660255 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:10:45.303 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.300+0000 7fd95d841700 1 --2- 172.21.15.176:0/2579660255 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9580fff00 0x7fd9580fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:10:45.303 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.300+0000 7fd95e042700 1 -- 172.21.15.176:0/2579660255 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9580fff00 msgr2=0x7fd9580fb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:10:45.304 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.300+0000 7fd95e042700 1 --2- 172.21.15.176:0/2579660255 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9580fff00 0x7fd9580fb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:10:45.304 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.300+0000 7fd95e042700 1 -- 172.21.15.176:0/2579660255 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd954005040 con 0x7fd9580662e0 2024-03-19T03:10:45.304 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.300+0000 7fd95e042700 1 --2- 172.21.15.176:0/2579660255 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9580662e0 0x7fd9580fd3f0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd95400b970 tx=0x7fd954009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:10:45.304 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.300+0000 7fd94affd700 1 -- 172.21.15.176:0/2579660255 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd95400e040 con 0x7fd9580662e0 2024-03-19T03:10:45.304 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.300+0000 7fd94affd700 1 -- 172.21.15.176:0/2579660255 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd954004290 con 0x7fd9580662e0 2024-03-19T03:10:45.304 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.300+0000 7fd9602a6700 1 -- 172.21.15.176:0/2579660255 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9580fc260 con 0x7fd9580662e0 2024-03-19T03:10:45.304 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.300+0000 7fd94affd700 1 -- 172.21.15.176:0/2579660255 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd95401c3f0 con 0x7fd9580662e0 2024-03-19T03:10:45.304 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.300+0000 7fd9602a6700 1 -- 172.21.15.176:0/2579660255 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd958115a70 con 0x7fd9580662e0 2024-03-19T03:10:45.305 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.301+0000 7fd9602a6700 1 -- 172.21.15.176:0/2579660255 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd95805f5f0 con 0x7fd9580662e0 2024-03-19T03:10:45.305 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.302+0000 7fd94affd700 1 -- 172.21.15.176:0/2579660255 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7fd954013420 con 0x7fd9580662e0 2024-03-19T03:10:45.305 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.302+0000 7fd94affd700 1 --2- 172.21.15.176:0/2579660255 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd944069210 0x7fd94406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:10:45.305 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.302+0000 7fd94affd700 1 -- 172.21.15.176:0/2579660255 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 2842+0+0 (secure 0 0 0) 0x7fd954087810 con 0x7fd9580662e0 2024-03-19T03:10:45.305 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.302+0000 7fd95d841700 1 --2- 172.21.15.176:0/2579660255 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd944069210 0x7fd94406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:10:45.305 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.303+0000 7fd95d841700 1 --2- 172.21.15.176:0/2579660255 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd944069210 0x7fd94406b6c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd94c0102e0 tx=0x7fd94c00b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:10:45.308 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.307+0000 7fd94affd700 1 -- 172.21.15.176:0/2579660255 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fd954054660 con 0x7fd9580662e0 2024-03-19T03:10:45.497 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:45.496+0000 7fd9602a6700 1 -- 172.21.15.176:0/2579660255 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9580f8f70 con 0x7fd944069210 2024-03-19T03:10:46.012 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:45 smithi176 ceph-mon[36450]: osdmap e20: 3 total, 3 up, 3 in 2024-03-19T03:10:46.013 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:45 smithi176 ceph-mon[36450]: pgmap v75: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:10:46.013 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:45 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-19T03:10:46.013 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:45 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:10:46.013 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:45 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:10:46.013 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:45 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:10:46.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:45 smithi059 ceph-mon[29133]: osdmap e20: 3 total, 3 up, 3 in 2024-03-19T03:10:46.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:45 smithi059 ceph-mon[29133]: pgmap v75: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:10:46.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:45 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-19T03:10:46.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:45 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:10:46.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:45 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:10:46.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:45 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:10:46.833 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:46 smithi059 ceph-mon[29133]: from='client.24133 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:10:46.833 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-19T03:10:46.834 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:46 smithi059 ceph-mon[29133]: osdmap e21: 3 total, 3 up, 3 in 2024-03-19T03:10:46.834 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T03:10:46.834 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:10:47.044 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:46 smithi176 ceph-mon[36450]: from='client.24133 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:10:47.044 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:46 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-19T03:10:47.044 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:46 smithi176 ceph-mon[36450]: osdmap e21: 3 total, 3 up, 3 in 2024-03-19T03:10:47.044 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:46 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T03:10:47.045 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:46 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:10:47.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:10:47 smithi059 sudo[66920]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T03:10:47.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:10:47 smithi059 sudo[66920]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T03:10:47.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:10:47 smithi059 sudo[66920]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T03:10:47.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:10:47 smithi059 sudo[66920]: pam_unix(sudo:session): session closed for user root 2024-03-19T03:10:47.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:47 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-19T03:10:47.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:47 smithi059 ceph-mon[29133]: osdmap e22: 3 total, 3 up, 3 in 2024-03-19T03:10:47.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:47 smithi059 ceph-mon[29133]: Detected new or changed devices on smithi059 2024-03-19T03:10:47.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:47 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:47.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:47 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:47.841 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:47 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:10:47.841 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:47 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:10:47.841 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:47 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:10:47.841 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:47 smithi059 ceph-mon[29133]: Adjusting osd_memory_target on smithi059 to 4289M 2024-03-19T03:10:47.841 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:47 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:47.841 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:47 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:10:47.841 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:47 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:10:47.841 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:47 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:47.841 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:47 smithi059 ceph-mon[29133]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:10:47.841 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:47 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:10:47.899 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:10:47.897+0000 7fd94affd700 1 -- 172.21.15.176:0/2579660255 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd954054a30 con 0x7fd9580662e0 2024-03-19T03:10:48.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:47 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-19T03:10:48.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:47 smithi176 ceph-mon[36450]: osdmap e22: 3 total, 3 up, 3 in 2024-03-19T03:10:48.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:47 smithi176 ceph-mon[36450]: Detected new or changed devices on smithi059 2024-03-19T03:10:48.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:47 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:48.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:47 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:48.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:47 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:10:48.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:47 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:10:48.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:47 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:10:48.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:47 smithi176 ceph-mon[36450]: Adjusting osd_memory_target on smithi059 to 4289M 2024-03-19T03:10:48.130 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:47 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:48.130 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:47 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:10:48.130 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:47 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:10:48.130 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:47 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:48.130 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:47 smithi176 ceph-mon[36450]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:10:48.130 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:47 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:10:48.203 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:10:47 smithi059 sudo[66955]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T03:10:48.204 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:10:47 smithi059 sudo[66955]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T03:10:48.204 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:10:47 smithi059 sudo[66955]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T03:10:48.204 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:10:47 smithi059 sudo[66955]: pam_unix(sudo:session): session closed for user root 2024-03-19T03:10:48.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:10:48 smithi059 sudo[67007]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T03:10:48.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:10:48 smithi059 sudo[67007]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T03:10:48.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:10:48 smithi059 sudo[67007]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T03:10:48.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:10:48 smithi059 sudo[67007]: pam_unix(sudo:session): session closed for user root 2024-03-19T03:10:48.728 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:10:48 smithi059 sudo[67032]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T03:10:48.729 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:10:48 smithi059 sudo[67032]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T03:10:48.729 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:10:48 smithi059 sudo[67032]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T03:10:48.729 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:10:48 smithi059 sudo[67032]: pam_unix(sudo:session): session closed for user root 2024-03-19T03:10:49.013 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:10:48 smithi059 sudo[67082]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T03:10:49.013 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:10:48 smithi059 sudo[67082]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T03:10:49.013 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:10:48 smithi059 sudo[67082]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T03:10:49.013 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:10:48 smithi059 sudo[67082]: pam_unix(sudo:session): session closed for user root 2024-03-19T03:10:49.013 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:48 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/1232251663' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3cf9095-7ce2-4d0f-bc65-2cda09504d15"}]: dispatch 2024-03-19T03:10:49.013 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:48 smithi059 ceph-mon[29133]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3cf9095-7ce2-4d0f-bc65-2cda09504d15"}]: dispatch 2024-03-19T03:10:49.013 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:48 smithi059 ceph-mon[29133]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3cf9095-7ce2-4d0f-bc65-2cda09504d15"}]': finished 2024-03-19T03:10:49.013 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:48 smithi059 ceph-mon[29133]: osdmap e23: 4 total, 3 up, 4 in 2024-03-19T03:10:49.013 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:48 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:10:49.013 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:48 smithi059 ceph-mon[29133]: mgrmap e19: smithi059.azpecc(active, since 2m), standbys: smithi176.ycmqbw 2024-03-19T03:10:49.014 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:48 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:49.014 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:48 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/1702094181' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:10:49.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:48 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/1232251663' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3cf9095-7ce2-4d0f-bc65-2cda09504d15"}]: dispatch 2024-03-19T03:10:49.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:48 smithi176 ceph-mon[36450]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3cf9095-7ce2-4d0f-bc65-2cda09504d15"}]: dispatch 2024-03-19T03:10:49.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:48 smithi176 ceph-mon[36450]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3cf9095-7ce2-4d0f-bc65-2cda09504d15"}]': finished 2024-03-19T03:10:49.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:48 smithi176 ceph-mon[36450]: osdmap e23: 4 total, 3 up, 4 in 2024-03-19T03:10:49.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:48 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:10:49.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:48 smithi176 ceph-mon[36450]: mgrmap e19: smithi059.azpecc(active, since 2m), standbys: smithi176.ycmqbw 2024-03-19T03:10:49.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:48 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:49.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:48 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/1702094181' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:10:49.274 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:10:49 smithi059 sudo[67109]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T03:10:49.274 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:10:49 smithi059 sudo[67109]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T03:10:49.274 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:10:49 smithi059 sudo[67109]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T03:10:49.274 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:10:49 smithi059 sudo[67109]: pam_unix(sudo:session): session closed for user root 2024-03-19T03:10:49.545 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:49 smithi059 sudo[67115]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T03:10:49.545 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:49 smithi059 sudo[67115]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T03:10:49.545 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:49 smithi059 sudo[67115]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T03:10:49.545 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:49 smithi059 sudo[67115]: pam_unix(sudo:session): session closed for user root 2024-03-19T03:10:49.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:49 smithi059 sudo[67164]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T03:10:49.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:49 smithi059 sudo[67164]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T03:10:49.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:49 smithi059 sudo[67164]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T03:10:49.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:49 smithi059 sudo[67164]: pam_unix(sudo:session): session closed for user root 2024-03-19T03:10:50.053 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:49 smithi176 sudo[40388]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T03:10:50.054 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:49 smithi176 sudo[40388]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T03:10:50.054 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:49 smithi176 sudo[40388]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T03:10:50.054 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:49 smithi176 sudo[40388]: pam_unix(sudo:session): session closed for user root 2024-03-19T03:10:50.315 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:50 smithi059 ceph-mon[29133]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:10:50.315 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:50 smithi059 ceph-mon[29133]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T03:10:50.315 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:50 smithi059 ceph-mon[29133]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T03:10:50.316 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:50 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-03-19T03:10:50.316 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:50 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi176"}]: dispatch 2024-03-19T03:10:50.316 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:50 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-03-19T03:10:50.316 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:50 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi176"}]: dispatch 2024-03-19T03:10:50.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:50 smithi176 sudo[40392]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T03:10:50.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:50 smithi176 sudo[40392]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T03:10:50.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:50 smithi176 sudo[40392]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T03:10:50.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:50 smithi176 sudo[40392]: pam_unix(sudo:session): session closed for user root 2024-03-19T03:10:50.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:50 smithi176 ceph-mon[36450]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:10:50.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:50 smithi176 ceph-mon[36450]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T03:10:50.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:50 smithi176 ceph-mon[36450]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T03:10:50.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:50 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-03-19T03:10:50.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:50 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi176"}]: dispatch 2024-03-19T03:10:50.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:50 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-03-19T03:10:50.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:50 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mon metadata", "id": "smithi176"}]: dispatch 2024-03-19T03:10:51.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:51 smithi059 ceph-mon[29133]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T03:10:51.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:51 smithi059 ceph-mon[29133]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T03:10:51.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:51 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:51.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:51 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:51.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:51 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:51.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:51 smithi176 ceph-mon[36450]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T03:10:51.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:51 smithi176 ceph-mon[36450]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T03:10:51.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:51 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:51.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:51 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:51.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:51 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:52.332 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:52 smithi176 ceph-mon[36450]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:10:52.332 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:52 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:10:52.332 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:52 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:10:52.332 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:52 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:52.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:52 smithi059 ceph-mon[29133]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:10:52.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:52 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:10:52.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:52 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:10:52.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:52 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:10:54.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:54 smithi176 ceph-mon[36450]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:10:54.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:54 smithi059 ceph-mon[29133]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:10:55.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:55 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-19T03:10:55.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:55 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:10:55.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-19T03:10:55.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:55 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:10:56.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:56 smithi059 ceph-mon[29133]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:10:56.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:56 smithi059 ceph-mon[29133]: Deploying daemon osd.3 on smithi176 2024-03-19T03:10:56.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:56 smithi176 ceph-mon[36450]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:10:56.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:56 smithi176 ceph-mon[36450]: Deploying daemon osd.3 on smithi176 2024-03-19T03:10:58.337 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:10:58 smithi176 ceph-mon[36450]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:10:58.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:10:58 smithi059 ceph-mon[29133]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:11:00.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:00 smithi059 ceph-mon[29133]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:11:00.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:00 smithi176 ceph-mon[36450]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:11:01.415 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:01.415 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:01.415 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:11:01.415 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:11:01.524 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:01.524 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:01.524 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:11:01.524 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:11:02.365 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:02 smithi176 ceph-mon[36450]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:11:02.477 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:02 smithi059 ceph-mon[29133]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:11:04.203 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:04 smithi176 ceph-mon[36450]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:11:04.203 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:04 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:04.203 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:04 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:04.203 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:04 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:11:04.203 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:04 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:11:04.203 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:04 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:04.203 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:04 smithi176 ceph-mon[36450]: from='osd.3 [v2:172.21.15.176:6800/3291292674,v1:172.21.15.176:6801/3291292674]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-19T03:11:04.203 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:04 smithi176 ceph-mon[36450]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-19T03:11:04.261 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:04.260+0000 7fd94affd700 1 -- 172.21.15.176:0/2579660255 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd9580f8f70 con 0x7fd944069210 2024-03-19T03:11:04.262 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 3 on host 'smithi176' 2024-03-19T03:11:04.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:04.262+0000 7fd9602a6700 1 -- 172.21.15.176:0/2579660255 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd944069210 msgr2=0x7fd94406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:11:04.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:04.262+0000 7fd9602a6700 1 --2- 172.21.15.176:0/2579660255 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd944069210 0x7fd94406b6c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd94c0102e0 tx=0x7fd94c00b040 comp rx=0 tx=0).stop 2024-03-19T03:11:04.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:04.262+0000 7fd9602a6700 1 -- 172.21.15.176:0/2579660255 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9580662e0 msgr2=0x7fd9580fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:11:04.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:04.262+0000 7fd9602a6700 1 --2- 172.21.15.176:0/2579660255 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9580662e0 0x7fd9580fd3f0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd95400b970 tx=0x7fd954009e30 comp rx=0 tx=0).stop 2024-03-19T03:11:04.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:04.262+0000 7fd9602a6700 1 -- 172.21.15.176:0/2579660255 shutdown_connections 2024-03-19T03:11:04.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:04.262+0000 7fd9602a6700 1 --2- 172.21.15.176:0/2579660255 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd944069210 0x7fd94406b6c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:11:04.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:04.262+0000 7fd9602a6700 1 --2- 172.21.15.176:0/2579660255 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9580662e0 0x7fd9580fd3f0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:11:04.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:04.262+0000 7fd9602a6700 1 --2- 172.21.15.176:0/2579660255 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9580fff00 0x7fd9580fb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:11:04.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:04.262+0000 7fd9602a6700 1 -- 172.21.15.176:0/2579660255 >> 172.21.15.176:0/2579660255 conn(0x7fd9580f7430 msgr2=0x7fd9580f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:11:04.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:04.263+0000 7fd9602a6700 1 -- 172.21.15.176:0/2579660255 shutdown_connections 2024-03-19T03:11:04.265 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:04.263+0000 7fd9602a6700 1 -- 172.21.15.176:0/2579660255 wait complete. 2024-03-19T03:11:04.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:04 smithi059 ceph-mon[29133]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:11:04.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:04 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:04.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:04 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:04.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:04 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:11:04.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:04 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:11:04.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:04 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:04.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:04 smithi059 ceph-mon[29133]: from='osd.3 [v2:172.21.15.176:6800/3291292674,v1:172.21.15.176:6801/3291292674]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-19T03:11:04.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:04 smithi059 ceph-mon[29133]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-19T03:11:04.856 DEBUG:teuthology.orchestra.run.smithi176:osd.3> sudo journalctl -f -n 0 -u ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@osd.3.service 2024-03-19T03:11:04.858 INFO:tasks.cephadm:Deploying osd.4 on smithi176 with /dev/vg_nvme/lv_3... 2024-03-19T03:11:04.858 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-19T03:11:05.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:05 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:05.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:05 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:05.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:05 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:11:05.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:05 smithi176 ceph-mon[36450]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-19T03:11:05.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:05 smithi176 ceph-mon[36450]: osdmap e24: 4 total, 3 up, 4 in 2024-03-19T03:11:05.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:05 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:11:05.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:05 smithi176 ceph-mon[36450]: from='osd.3 [v2:172.21.15.176:6800/3291292674,v1:172.21.15.176:6801/3291292674]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-19T03:11:05.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:05 smithi176 ceph-mon[36450]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-19T03:11:05.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:05 smithi176 ceph-mon[36450]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:11:05.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:05 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:05.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:05 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:05.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:05 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:11:05.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:05 smithi059 ceph-mon[29133]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-19T03:11:05.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:05 smithi059 ceph-mon[29133]: osdmap e24: 4 total, 3 up, 4 in 2024-03-19T03:11:05.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:05 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:11:05.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:05 smithi059 ceph-mon[29133]: from='osd.3 [v2:172.21.15.176:6800/3291292674,v1:172.21.15.176:6801/3291292674]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-19T03:11:05.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:05 smithi059 ceph-mon[29133]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-19T03:11:05.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:05 smithi059 ceph-mon[29133]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:11:05.684 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi176/config 2024-03-19T03:11:05.787 INFO:journalctl@ceph.osd.3.smithi176.stdout:Mar 19 03:11:05 smithi176 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-3[42016]: 2024-03-19T03:11:05.405+0000 7f2656176700 -1 osd.3 0 waiting for initial osdmap 2024-03-19T03:11:05.787 INFO:journalctl@ceph.osd.3.smithi176.stdout:Mar 19 03:11:05 smithi176 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-3[42016]: 2024-03-19T03:11:05.410+0000 7f2650f71700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T03:11:06.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:06 smithi176 ceph-mon[36450]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-03-19T03:11:06.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:06 smithi176 ceph-mon[36450]: osdmap e25: 4 total, 3 up, 4 in 2024-03-19T03:11:06.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:06 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:11:06.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:06 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:11:06.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:06 smithi059 ceph-mon[29133]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-03-19T03:11:06.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:06 smithi059 ceph-mon[29133]: osdmap e25: 4 total, 3 up, 4 in 2024-03-19T03:11:06.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:06 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:11:06.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:06 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:11:07.524 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:07 smithi176 ceph-mon[36450]: purged_snaps scrub starts 2024-03-19T03:11:07.524 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:07 smithi176 ceph-mon[36450]: purged_snaps scrub ok 2024-03-19T03:11:07.524 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:07 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:11:07.524 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:07 smithi176 ceph-mon[36450]: osd.3 [v2:172.21.15.176:6800/3291292674,v1:172.21.15.176:6801/3291292674] boot 2024-03-19T03:11:07.524 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:07 smithi176 ceph-mon[36450]: osdmap e26: 4 total, 4 up, 4 in 2024-03-19T03:11:07.524 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:07 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:11:07.524 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:07 smithi176 ceph-mon[36450]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:11:07.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:07 smithi059 ceph-mon[29133]: purged_snaps scrub starts 2024-03-19T03:11:07.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:07 smithi059 ceph-mon[29133]: purged_snaps scrub ok 2024-03-19T03:11:07.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:07 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:11:07.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:07 smithi059 ceph-mon[29133]: osd.3 [v2:172.21.15.176:6800/3291292674,v1:172.21.15.176:6801/3291292674] boot 2024-03-19T03:11:07.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:07 smithi059 ceph-mon[29133]: osdmap e26: 4 total, 4 up, 4 in 2024-03-19T03:11:07.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:07 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:11:07.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:07 smithi059 ceph-mon[29133]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-19T03:11:08.541 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:08 smithi176 ceph-mon[36450]: osdmap e27: 4 total, 4 up, 4 in 2024-03-19T03:11:08.541 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:08 smithi176 ceph-mon[36450]: Detected new or changed devices on smithi176 2024-03-19T03:11:08.541 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:08 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:08.541 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:08 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:08.541 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:08 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:11:08.541 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:08 smithi176 ceph-mon[36450]: Adjusting osd_memory_target on smithi176 to 15936M 2024-03-19T03:11:08.541 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:08 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:08.541 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:08 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:11:08.541 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:08 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:11:08.541 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:08 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:08.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:08 smithi059 ceph-mon[29133]: osdmap e27: 4 total, 4 up, 4 in 2024-03-19T03:11:08.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:08 smithi059 ceph-mon[29133]: Detected new or changed devices on smithi176 2024-03-19T03:11:08.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:08 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:08.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:08 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:08.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:08 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:11:08.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:08 smithi059 ceph-mon[29133]: Adjusting osd_memory_target on smithi176 to 15936M 2024-03-19T03:11:08.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:08 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:08.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:08 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:11:08.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:08 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:11:08.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:08 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:09.072 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-19T03:11:09.072 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-19T03:11:09.072 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2024-03-19T03:11:09.072 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2024-03-19T03:11:09.072 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113357 s, 925 MB/s 2024-03-19T03:11:09.072 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2024-03-19T03:11:09.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:09 smithi059 ceph-mon[29133]: osdmap e28: 4 total, 4 up, 4 in 2024-03-19T03:11:09.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:09 smithi059 ceph-mon[29133]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-19T03:11:09.674 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi176:vg_nvme/lv_3 2024-03-19T03:11:09.700 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:09 smithi176 ceph-mon[36450]: osdmap e28: 4 total, 4 up, 4 in 2024-03-19T03:11:09.700 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:09 smithi176 ceph-mon[36450]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-19T03:11:09.925 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi176/config 2024-03-19T03:11:10.720 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:10 smithi176 ceph-mon[36450]: osdmap e29: 4 total, 4 up, 4 in 2024-03-19T03:11:10.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:10 smithi059 ceph-mon[29133]: osdmap e29: 4 total, 4 up, 4 in 2024-03-19T03:11:11.475 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.472+0000 7f36451e4700 1 -- 172.21.15.176:0/2121142783 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3640066eb0 msgr2=0x7f36400ffa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:11:11.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.472+0000 7f36451e4700 1 --2- 172.21.15.176:0/2121142783 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3640066eb0 0x7f36400ffa80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3630004940 tx=0x7f3630009e30 comp rx=0 tx=0).stop 2024-03-19T03:11:11.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.472+0000 7f36451e4700 1 -- 172.21.15.176:0/2121142783 shutdown_connections 2024-03-19T03:11:11.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.472+0000 7f36451e4700 1 --2- 172.21.15.176:0/2121142783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36400fffc0 0x7f36401023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:11:11.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.472+0000 7f36451e4700 1 --2- 172.21.15.176:0/2121142783 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3640066eb0 0x7f36400ffa80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:11:11.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.472+0000 7f36451e4700 1 -- 172.21.15.176:0/2121142783 >> 172.21.15.176:0/2121142783 conn(0x7f36400f7430 msgr2=0x7f36400f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:11:11.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.472+0000 7f36451e4700 1 -- 172.21.15.176:0/2121142783 shutdown_connections 2024-03-19T03:11:11.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.472+0000 7f36451e4700 1 -- 172.21.15.176:0/2121142783 wait complete. 2024-03-19T03:11:11.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.473+0000 7f36451e4700 1 Processor -- start 2024-03-19T03:11:11.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.473+0000 7f36451e4700 1 -- start start 2024-03-19T03:11:11.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.473+0000 7f36451e4700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3640066eb0 0x7f364010d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:11:11.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.473+0000 7f36451e4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36400fffc0 0x7f364010def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:11:11.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.473+0000 7f36451e4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364010e4f0 con 0x7f36400fffc0 2024-03-19T03:11:11.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.473+0000 7f36451e4700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364010e660 con 0x7f3640066eb0 2024-03-19T03:11:11.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.473+0000 7f363ed9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3640066eb0 0x7f364010d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:11:11.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.473+0000 7f363ed9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3640066eb0 0x7f364010d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:43336/0 (socket says 172.21.15.176:43336) 2024-03-19T03:11:11.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.473+0000 7f363ed9d700 1 -- 172.21.15.176:0/3133551896 learned_addr learned my addr 172.21.15.176:0/3133551896 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:11:11.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.473+0000 7f363e59c700 1 --2- 172.21.15.176:0/3133551896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36400fffc0 0x7f364010def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:11:11.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.474+0000 7f363ed9d700 1 -- 172.21.15.176:0/3133551896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36400fffc0 msgr2=0x7f364010def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:11:11.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.474+0000 7f363ed9d700 1 --2- 172.21.15.176:0/3133551896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36400fffc0 0x7f364010def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:11:11.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.474+0000 7f363ed9d700 1 -- 172.21.15.176:0/3133551896 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3630005040 con 0x7f3640066eb0 2024-03-19T03:11:11.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.474+0000 7f363ed9d700 1 --2- 172.21.15.176:0/3133551896 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3640066eb0 0x7f364010d9b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f363000b3b0 tx=0x7f3630009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:11:11.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.474+0000 7f36277fe700 1 -- 172.21.15.176:0/3133551896 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f363000e040 con 0x7f3640066eb0 2024-03-19T03:11:11.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.474+0000 7f36277fe700 1 -- 172.21.15.176:0/3133551896 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f363000bbf0 con 0x7f3640066eb0 2024-03-19T03:11:11.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.474+0000 7f36451e4700 1 -- 172.21.15.176:0/3133551896 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3640113060 con 0x7f3640066eb0 2024-03-19T03:11:11.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.474+0000 7f36277fe700 1 -- 172.21.15.176:0/3133551896 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f363001c630 con 0x7f3640066eb0 2024-03-19T03:11:11.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.474+0000 7f36451e4700 1 -- 172.21.15.176:0/3133551896 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3640113620 con 0x7f3640066eb0 2024-03-19T03:11:11.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.475+0000 7f36451e4700 1 -- 172.21.15.176:0/3133551896 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f364004c710 con 0x7f3640066eb0 2024-03-19T03:11:11.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.475+0000 7f36277fe700 1 -- 172.21.15.176:0/3133551896 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f363000bd60 con 0x7f3640066eb0 2024-03-19T03:11:11.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.476+0000 7f36277fe700 1 --2- 172.21.15.176:0/3133551896 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f36280692a0 0x7f362806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:11:11.479 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.476+0000 7f36277fe700 1 -- 172.21.15.176:0/3133551896 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3770+0+0 (secure 0 0 0) 0x7f36300877a0 con 0x7f3640066eb0 2024-03-19T03:11:11.479 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.476+0000 7f363e59c700 1 --2- 172.21.15.176:0/3133551896 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f36280692a0 0x7f362806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:11:11.479 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.476+0000 7f363e59c700 1 --2- 172.21.15.176:0/3133551896 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f36280692a0 0x7f362806b750 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f36340041d0 tx=0x7f36340043c0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:11:11.482 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.480+0000 7f36277fe700 1 -- 172.21.15.176:0/3133551896 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f3630057c00 con 0x7f3640066eb0 2024-03-19T03:11:11.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:11 smithi176 ceph-mon[36450]: osdmap e30: 4 total, 4 up, 4 in 2024-03-19T03:11:11.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:11 smithi176 ceph-mon[36450]: pgmap v98: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-19T03:11:11.672 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:11.670+0000 7f36451e4700 1 -- 172.21.15.176:0/3133551896 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f36400f8f70 con 0x7f36280692a0 2024-03-19T03:11:11.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:11 smithi059 ceph-mon[29133]: osdmap e30: 4 total, 4 up, 4 in 2024-03-19T03:11:11.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:11 smithi059 ceph-mon[29133]: pgmap v98: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-19T03:11:12.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:12 smithi176 ceph-mon[36450]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:11:12.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:12 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:11:12.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:12 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:11:12.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:12 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:11:12.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:12 smithi059 ceph-mon[29133]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:11:12.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:12 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:11:12.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:12 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:11:12.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:12 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:11:13.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:13 smithi176 ceph-mon[36450]: pgmap v99: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-19T03:11:13.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:13 smithi059 ceph-mon[29133]: pgmap v99: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-19T03:11:14.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:14 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/1845488630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b102d232-9f65-4d96-ad81-b3d1a0b8faf2"}]: dispatch 2024-03-19T03:11:14.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:14 smithi059 ceph-mon[29133]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b102d232-9f65-4d96-ad81-b3d1a0b8faf2"}]: dispatch 2024-03-19T03:11:14.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:14 smithi059 ceph-mon[29133]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b102d232-9f65-4d96-ad81-b3d1a0b8faf2"}]': finished 2024-03-19T03:11:14.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:14 smithi059 ceph-mon[29133]: osdmap e31: 5 total, 4 up, 5 in 2024-03-19T03:11:14.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:14 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:11:14.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:14 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/1845488630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b102d232-9f65-4d96-ad81-b3d1a0b8faf2"}]: dispatch 2024-03-19T03:11:14.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:14 smithi176 ceph-mon[36450]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b102d232-9f65-4d96-ad81-b3d1a0b8faf2"}]: dispatch 2024-03-19T03:11:14.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:14 smithi176 ceph-mon[36450]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b102d232-9f65-4d96-ad81-b3d1a0b8faf2"}]': finished 2024-03-19T03:11:14.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:14 smithi176 ceph-mon[36450]: osdmap e31: 5 total, 4 up, 5 in 2024-03-19T03:11:14.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:14 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:11:15.836 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:15 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/3632823896' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:11:15.837 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:15 smithi176 ceph-mon[36450]: pgmap v101: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T03:11:15.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:15 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/3632823896' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:11:15.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:15 smithi059 ceph-mon[29133]: pgmap v101: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T03:11:16.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:16 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:11:16.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:16 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:11:17.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:17 smithi059 ceph-mon[29133]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-19T03:11:17.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:17 smithi176 ceph-mon[36450]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-19T03:11:20.218 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:19 smithi176 ceph-mon[36450]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-19T03:11:20.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:19 smithi059 ceph-mon[29133]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-19T03:11:21.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:21 smithi059 ceph-mon[29133]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-19T03:11:21.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:21 smithi176 ceph-mon[36450]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-19T03:11:22.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:22 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-19T03:11:22.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:22 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:11:22.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:22 smithi176 ceph-mon[36450]: Deploying daemon osd.4 on smithi176 2024-03-19T03:11:22.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:22 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-19T03:11:22.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:22 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:11:22.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:22 smithi059 ceph-mon[29133]: Deploying daemon osd.4 on smithi176 2024-03-19T03:11:23.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:23 smithi059 ceph-mon[29133]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-19T03:11:23.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:23 smithi176 ceph-mon[36450]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-19T03:11:26.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:25 smithi176 ceph-mon[36450]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-19T03:11:26.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:25 smithi059 ceph-mon[29133]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-19T03:11:27.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:27 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:27.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:27 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:27.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:27 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:11:27.629 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:27 smithi176 ceph-mon[36450]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-19T03:11:27.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:27 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:27.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:27 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:27.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:27 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:11:27.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:27 smithi059 ceph-mon[29133]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-19T03:11:30.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:29 smithi176 ceph-mon[36450]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-19T03:11:30.239 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 4 on host 'smithi176' 2024-03-19T03:11:30.240 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:30.236+0000 7f36277fe700 1 -- 172.21.15.176:0/3133551896 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f36400f8f70 con 0x7f36280692a0 2024-03-19T03:11:30.242 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:30.240+0000 7f36451e4700 1 -- 172.21.15.176:0/3133551896 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f36280692a0 msgr2=0x7f362806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:11:30.242 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:30.240+0000 7f36451e4700 1 --2- 172.21.15.176:0/3133551896 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f36280692a0 0x7f362806b750 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f36340041d0 tx=0x7f36340043c0 comp rx=0 tx=0).stop 2024-03-19T03:11:30.242 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:30.240+0000 7f36451e4700 1 -- 172.21.15.176:0/3133551896 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3640066eb0 msgr2=0x7f364010d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:11:30.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:30.240+0000 7f36451e4700 1 --2- 172.21.15.176:0/3133551896 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3640066eb0 0x7f364010d9b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f363000b3b0 tx=0x7f3630009e30 comp rx=0 tx=0).stop 2024-03-19T03:11:30.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:30.240+0000 7f36451e4700 1 -- 172.21.15.176:0/3133551896 shutdown_connections 2024-03-19T03:11:30.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:30.240+0000 7f36451e4700 1 --2- 172.21.15.176:0/3133551896 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f36280692a0 0x7f362806b750 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:11:30.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:30.240+0000 7f36451e4700 1 --2- 172.21.15.176:0/3133551896 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3640066eb0 0x7f364010d9b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:11:30.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:30.240+0000 7f36451e4700 1 --2- 172.21.15.176:0/3133551896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36400fffc0 0x7f364010def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:11:30.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:30.240+0000 7f36451e4700 1 -- 172.21.15.176:0/3133551896 >> 172.21.15.176:0/3133551896 conn(0x7f36400f7430 msgr2=0x7f36400f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:11:30.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:30.240+0000 7f36451e4700 1 -- 172.21.15.176:0/3133551896 shutdown_connections 2024-03-19T03:11:30.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:30.240+0000 7f36451e4700 1 -- 172.21.15.176:0/3133551896 wait complete. 2024-03-19T03:11:30.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:29 smithi059 ceph-mon[29133]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-19T03:11:30.862 DEBUG:teuthology.orchestra.run.smithi176:osd.4> sudo journalctl -f -n 0 -u ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@osd.4.service 2024-03-19T03:11:30.904 INFO:tasks.cephadm:Deploying osd.5 on smithi176 with /dev/vg_nvme/lv_2... 2024-03-19T03:11:30.904 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-19T03:11:31.227 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi176/config 2024-03-19T03:11:31.475 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:31 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:31.475 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:31 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:31.476 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:31 smithi176 ceph-mon[36450]: from='osd.4 [v2:172.21.15.176:6808/1006246702,v1:172.21.15.176:6809/1006246702]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T03:11:31.476 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:31 smithi176 ceph-mon[36450]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T03:11:31.476 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:31 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:11:31.476 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:31 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:31.476 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:31 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:31.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:31.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:31.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:31 smithi059 ceph-mon[29133]: from='osd.4 [v2:172.21.15.176:6808/1006246702,v1:172.21.15.176:6809/1006246702]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T03:11:31.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:31 smithi059 ceph-mon[29133]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T03:11:31.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:11:31.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:31.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:32.479 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:32 smithi176 ceph-mon[36450]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-19T03:11:32.480 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:32 smithi176 ceph-mon[36450]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-19T03:11:32.480 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:32 smithi176 ceph-mon[36450]: osdmap e32: 5 total, 4 up, 5 in 2024-03-19T03:11:32.480 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:32 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:11:32.480 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:32 smithi176 ceph-mon[36450]: from='osd.4 [v2:172.21.15.176:6808/1006246702,v1:172.21.15.176:6809/1006246702]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-19T03:11:32.480 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:32 smithi176 ceph-mon[36450]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-19T03:11:32.480 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 19 03:11:32 smithi176 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-4[48463]: 2024-03-19T03:11:32.222+0000 7f765a1f5700 -1 osd.4 0 waiting for initial osdmap 2024-03-19T03:11:32.480 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 19 03:11:32 smithi176 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-4[48463]: 2024-03-19T03:11:32.229+0000 7f7654ff0700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T03:11:32.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:32 smithi059 ceph-mon[29133]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-19T03:11:32.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:32 smithi059 ceph-mon[29133]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-19T03:11:32.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:32 smithi059 ceph-mon[29133]: osdmap e32: 5 total, 4 up, 5 in 2024-03-19T03:11:32.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:32 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:11:32.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:32 smithi059 ceph-mon[29133]: from='osd.4 [v2:172.21.15.176:6808/1006246702,v1:172.21.15.176:6809/1006246702]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-19T03:11:32.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:32 smithi059 ceph-mon[29133]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-19T03:11:33.084 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-19T03:11:33.084 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-19T03:11:33.084 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2024-03-19T03:11:33.085 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2024-03-19T03:11:33.085 INFO:teuthology.orchestra.run.smithi176.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124395 s, 843 MB/s 2024-03-19T03:11:33.085 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2024-03-19T03:11:33.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:33 smithi176 ceph-mon[36450]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-03-19T03:11:33.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:33 smithi176 ceph-mon[36450]: osdmap e33: 5 total, 4 up, 5 in 2024-03-19T03:11:33.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:33 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:11:33.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:33 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:11:33.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:33 smithi176 ceph-mon[36450]: osd.4 [v2:172.21.15.176:6808/1006246702,v1:172.21.15.176:6809/1006246702] boot 2024-03-19T03:11:33.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:33 smithi059 ceph-mon[29133]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-03-19T03:11:33.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:33 smithi059 ceph-mon[29133]: osdmap e33: 5 total, 4 up, 5 in 2024-03-19T03:11:33.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:33 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:11:33.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:33 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:11:33.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:33 smithi059 ceph-mon[29133]: osd.4 [v2:172.21.15.176:6808/1006246702,v1:172.21.15.176:6809/1006246702] boot 2024-03-19T03:11:34.506 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:34 smithi176 ceph-mon[36450]: purged_snaps scrub starts 2024-03-19T03:11:34.506 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:34 smithi176 ceph-mon[36450]: purged_snaps scrub ok 2024-03-19T03:11:34.507 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:34 smithi176 ceph-mon[36450]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-19T03:11:34.507 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:34 smithi176 ceph-mon[36450]: osdmap e34: 5 total, 5 up, 5 in 2024-03-19T03:11:34.507 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:34 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:11:34.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:34 smithi059 ceph-mon[29133]: purged_snaps scrub starts 2024-03-19T03:11:34.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:34 smithi059 ceph-mon[29133]: purged_snaps scrub ok 2024-03-19T03:11:34.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:34 smithi059 ceph-mon[29133]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-19T03:11:34.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:34 smithi059 ceph-mon[29133]: osdmap e34: 5 total, 5 up, 5 in 2024-03-19T03:11:34.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:34 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:11:35.018 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi176:vg_nvme/lv_2 2024-03-19T03:11:35.841 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi176/config 2024-03-19T03:11:36.282 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:36 smithi176 ceph-mon[36450]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:36.283 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:36 smithi176 ceph-mon[36450]: osdmap e35: 5 total, 5 up, 5 in 2024-03-19T03:11:36.283 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:36 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:36.283 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:36 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:36.283 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:36 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:11:36.283 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:36 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:11:36.283 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:36 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:36.283 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:36 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:11:36.283 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:36 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:11:36.283 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:36 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:36.283 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:36 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:11:36.283 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:36 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:11:36.283 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:36 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:11:36.284 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:36 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:36.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:36 smithi059 ceph-mon[29133]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:36.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:36 smithi059 ceph-mon[29133]: osdmap e35: 5 total, 5 up, 5 in 2024-03-19T03:11:36.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:36 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:36.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:36 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:36.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:36 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:11:36.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:36 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:11:36.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:36 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:36.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:36 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:11:36.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:36 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:11:36.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:36 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:36.591 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:36 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:11:36.591 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:36 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:11:36.591 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:36 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:11:36.591 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:36 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:37.406 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.401+0000 7f9e4dc68700 1 -- 172.21.15.176:0/3621950086 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e480fcf80 msgr2=0x7f9e480fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:11:37.406 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.401+0000 7f9e4dc68700 1 --2- 172.21.15.176:0/3621950086 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e480fcf80 0x7f9e480fd3f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c004940 tx=0x7f9e3c009e30 comp rx=0 tx=0).stop 2024-03-19T03:11:37.406 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.401+0000 7f9e4dc68700 1 -- 172.21.15.176:0/3621950086 shutdown_connections 2024-03-19T03:11:37.406 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.401+0000 7f9e4dc68700 1 --2- 172.21.15.176:0/3621950086 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e480fcf80 0x7f9e480fd3f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:11:37.406 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.401+0000 7f9e4dc68700 1 --2- 172.21.15.176:0/3621950086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e480fbcd0 0x7f9e480fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:11:37.406 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.401+0000 7f9e4dc68700 1 -- 172.21.15.176:0/3621950086 >> 172.21.15.176:0/3621950086 conn(0x7f9e480f7450 msgr2=0x7f9e480f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:11:37.407 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.401+0000 7f9e4dc68700 1 -- 172.21.15.176:0/3621950086 shutdown_connections 2024-03-19T03:11:37.407 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.401+0000 7f9e4dc68700 1 -- 172.21.15.176:0/3621950086 wait complete. 2024-03-19T03:11:37.407 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.402+0000 7f9e4dc68700 1 Processor -- start 2024-03-19T03:11:37.407 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.402+0000 7f9e4dc68700 1 -- start start 2024-03-19T03:11:37.407 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.402+0000 7f9e4dc68700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e480fbcd0 0x7f9e48066cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:11:37.407 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.402+0000 7f9e4dc68700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e480fcf80 0x7f9e48112740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:11:37.407 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.402+0000 7f9e4dc68700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e48112c80 con 0x7f9e480fcf80 2024-03-19T03:11:37.407 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.402+0000 7f9e4dc68700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e48112df0 con 0x7f9e480fbcd0 2024-03-19T03:11:37.407 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.402+0000 7f9e477fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e480fbcd0 0x7f9e48066cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:11:37.407 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.402+0000 7f9e477fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e480fbcd0 0x7f9e48066cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:53714/0 (socket says 172.21.15.176:53714) 2024-03-19T03:11:37.407 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.402+0000 7f9e477fe700 1 -- 172.21.15.176:0/2470261369 learned_addr learned my addr 172.21.15.176:0/2470261369 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:11:37.407 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.403+0000 7f9e46ffd700 1 --2- 172.21.15.176:0/2470261369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e480fcf80 0x7f9e48112740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:11:37.407 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.403+0000 7f9e477fe700 1 -- 172.21.15.176:0/2470261369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e480fcf80 msgr2=0x7f9e48112740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:11:37.407 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.403+0000 7f9e477fe700 1 --2- 172.21.15.176:0/2470261369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e480fcf80 0x7f9e48112740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:11:37.408 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.403+0000 7f9e477fe700 1 -- 172.21.15.176:0/2470261369 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e3c005040 con 0x7f9e480fbcd0 2024-03-19T03:11:37.408 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.403+0000 7f9e477fe700 1 --2- 172.21.15.176:0/2470261369 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e480fbcd0 0x7f9e48066cd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9e3800b790 tx=0x7f9e38009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:11:37.408 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.403+0000 7f9e2ffff700 1 -- 172.21.15.176:0/2470261369 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e3800f770 con 0x7f9e480fbcd0 2024-03-19T03:11:37.408 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.403+0000 7f9e4dc68700 1 -- 172.21.15.176:0/2470261369 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e481130d0 con 0x7f9e480fbcd0 2024-03-19T03:11:37.408 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.403+0000 7f9e2ffff700 1 -- 172.21.15.176:0/2470261369 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e3800fda0 con 0x7f9e480fbcd0 2024-03-19T03:11:37.408 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.403+0000 7f9e2ffff700 1 -- 172.21.15.176:0/2470261369 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e3800d310 con 0x7f9e480fbcd0 2024-03-19T03:11:37.408 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.403+0000 7f9e4dc68700 1 -- 172.21.15.176:0/2470261369 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e48113690 con 0x7f9e480fbcd0 2024-03-19T03:11:37.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.404+0000 7f9e4dc68700 1 -- 172.21.15.176:0/2470261369 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e4804c710 con 0x7f9e480fbcd0 2024-03-19T03:11:37.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.405+0000 7f9e2ffff700 1 -- 172.21.15.176:0/2470261369 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f9e38010420 con 0x7f9e480fbcd0 2024-03-19T03:11:37.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.405+0000 7f9e2ffff700 1 --2- 172.21.15.176:0/2470261369 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f9e300692a0 0x7f9e3006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:11:37.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.405+0000 7f9e2ffff700 1 -- 172.21.15.176:0/2470261369 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f9e38086cc0 con 0x7f9e480fbcd0 2024-03-19T03:11:37.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.405+0000 7f9e46ffd700 1 --2- 172.21.15.176:0/2470261369 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f9e300692a0 0x7f9e3006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:11:37.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.406+0000 7f9e46ffd700 1 --2- 172.21.15.176:0/2470261369 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f9e300692a0 0x7f9e3006b750 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c00b3b0 tx=0x7f9e3c009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:11:37.412 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.410+0000 7f9e2ffff700 1 -- 172.21.15.176:0/2470261369 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f9e38056f70 con 0x7f9e480fbcd0 2024-03-19T03:11:37.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:37 smithi059 ceph-mon[29133]: Detected new or changed devices on smithi176 2024-03-19T03:11:37.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:37 smithi059 ceph-mon[29133]: Adjusting osd_memory_target on smithi176 to 7968M 2024-03-19T03:11:37.601 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:37.599+0000 7f9e4dc68700 1 -- 172.21.15.176:0/2470261369 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e48101240 con 0x7f9e300692a0 2024-03-19T03:11:37.602 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:37 smithi176 ceph-mon[36450]: Detected new or changed devices on smithi176 2024-03-19T03:11:37.602 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:37 smithi176 ceph-mon[36450]: Adjusting osd_memory_target on smithi176 to 7968M 2024-03-19T03:11:38.358 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:38 smithi176 ceph-mon[36450]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:38.359 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:38 smithi176 ceph-mon[36450]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:11:38.359 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:11:38.359 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:11:38.359 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:11:38.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:38 smithi059 ceph-mon[29133]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:38.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:38 smithi059 ceph-mon[29133]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:11:38.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:11:38.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:11:38.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:11:39.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:39 smithi059 ceph-mon[29133]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:39.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:39 smithi176 ceph-mon[36450]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:40.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:40 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/3824260669' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9c805ed-8aa0-4f39-a78f-c6a4b018eb7c"}]: dispatch 2024-03-19T03:11:40.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:40 smithi059 ceph-mon[29133]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9c805ed-8aa0-4f39-a78f-c6a4b018eb7c"}]: dispatch 2024-03-19T03:11:40.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:40 smithi059 ceph-mon[29133]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9c805ed-8aa0-4f39-a78f-c6a4b018eb7c"}]': finished 2024-03-19T03:11:40.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:40 smithi059 ceph-mon[29133]: osdmap e36: 6 total, 5 up, 6 in 2024-03-19T03:11:40.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:40 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:11:40.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:40 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/3824260669' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9c805ed-8aa0-4f39-a78f-c6a4b018eb7c"}]: dispatch 2024-03-19T03:11:40.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:40 smithi176 ceph-mon[36450]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9c805ed-8aa0-4f39-a78f-c6a4b018eb7c"}]: dispatch 2024-03-19T03:11:40.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:40 smithi176 ceph-mon[36450]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9c805ed-8aa0-4f39-a78f-c6a4b018eb7c"}]': finished 2024-03-19T03:11:40.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:40 smithi176 ceph-mon[36450]: osdmap e36: 6 total, 5 up, 6 in 2024-03-19T03:11:40.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:40 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:11:41.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:41 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/801060769' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:11:41.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:41 smithi059 ceph-mon[29133]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:41.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:41 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/801060769' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:11:41.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:41 smithi176 ceph-mon[36450]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:44.227 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:43 smithi176 ceph-mon[36450]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:44.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:43 smithi059 ceph-mon[29133]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:46.249 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:45 smithi176 ceph-mon[36450]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:46.249 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:45 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:11:46.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:45 smithi059 ceph-mon[29133]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:46.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:45 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:11:48.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:47 smithi176 ceph-mon[36450]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:48.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:47 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-19T03:11:48.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:47 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:11:48.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:47 smithi059 ceph-mon[29133]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:48.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:47 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-19T03:11:48.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:47 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:11:49.286 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:49 smithi176 ceph-mon[36450]: Deploying daemon osd.5 on smithi176 2024-03-19T03:11:49.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:49 smithi059 ceph-mon[29133]: Deploying daemon osd.5 on smithi176 2024-03-19T03:11:50.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:50 smithi176 ceph-mon[36450]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:50.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:50 smithi059 ceph-mon[29133]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:51.526 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:51 smithi176 ceph-mon[36450]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:51.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:51 smithi059 ceph-mon[29133]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:53.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:53 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:53.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:53 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:53.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:53 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:11:53.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:53 smithi059 ceph-mon[29133]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:53.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:53 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:53.629 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:53 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:53.629 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:53 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:11:53.629 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:53 smithi176 ceph-mon[36450]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:55.214 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:55.213+0000 7f9e2ffff700 1 -- 172.21.15.176:0/2470261369 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9e48101240 con 0x7f9e300692a0 2024-03-19T03:11:55.215 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 5 on host 'smithi176' 2024-03-19T03:11:55.216 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:55.215+0000 7f9e4dc68700 1 -- 172.21.15.176:0/2470261369 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f9e300692a0 msgr2=0x7f9e3006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:11:55.217 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:55.215+0000 7f9e4dc68700 1 --2- 172.21.15.176:0/2470261369 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f9e300692a0 0x7f9e3006b750 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c00b3b0 tx=0x7f9e3c009e30 comp rx=0 tx=0).stop 2024-03-19T03:11:55.217 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:55.215+0000 7f9e4dc68700 1 -- 172.21.15.176:0/2470261369 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e480fbcd0 msgr2=0x7f9e48066cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:11:55.217 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:55.215+0000 7f9e4dc68700 1 --2- 172.21.15.176:0/2470261369 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e480fbcd0 0x7f9e48066cd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9e3800b790 tx=0x7f9e38009250 comp rx=0 tx=0).stop 2024-03-19T03:11:55.217 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:55.215+0000 7f9e4dc68700 1 -- 172.21.15.176:0/2470261369 shutdown_connections 2024-03-19T03:11:55.217 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:55.215+0000 7f9e4dc68700 1 --2- 172.21.15.176:0/2470261369 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f9e300692a0 0x7f9e3006b750 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:11:55.217 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:55.215+0000 7f9e4dc68700 1 --2- 172.21.15.176:0/2470261369 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e480fbcd0 0x7f9e48066cd0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:11:55.217 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:55.215+0000 7f9e4dc68700 1 --2- 172.21.15.176:0/2470261369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e480fcf80 0x7f9e48112740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:11:55.217 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:55.215+0000 7f9e4dc68700 1 -- 172.21.15.176:0/2470261369 >> 172.21.15.176:0/2470261369 conn(0x7f9e480f7450 msgr2=0x7f9e480f8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:11:55.217 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:55.215+0000 7f9e4dc68700 1 -- 172.21.15.176:0/2470261369 shutdown_connections 2024-03-19T03:11:55.217 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:11:55.215+0000 7f9e4dc68700 1 -- 172.21.15.176:0/2470261369 wait complete. 2024-03-19T03:11:55.795 DEBUG:teuthology.orchestra.run.smithi176:osd.5> sudo journalctl -f -n 0 -u ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@osd.5.service 2024-03-19T03:11:55.798 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-03-19T03:11:55.799 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-19T03:11:56.071 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:11:56.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:56 smithi176 ceph-mon[36450]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:56.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:56 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:56.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:56 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:56.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:56 smithi176 ceph-mon[36450]: from='osd.5 [v2:172.21.15.176:6816/3279585260,v1:172.21.15.176:6817/3279585260]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T03:11:56.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:56 smithi176 ceph-mon[36450]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T03:11:56.380 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 19 03:11:56 smithi176 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-5[55356]: 2024-03-19T03:11:56.182+0000 7f681d4b1540 -1 osd.5 0 log_to_monitors true 2024-03-19T03:11:56.477 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:56 smithi059 ceph-mon[29133]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:56.477 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:56 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:56.477 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:56 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:56.477 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:56 smithi059 ceph-mon[29133]: from='osd.5 [v2:172.21.15.176:6816/3279585260,v1:172.21.15.176:6817/3279585260]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T03:11:56.477 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:56 smithi059 ceph-mon[29133]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T03:11:57.512 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:57 smithi059 ceph-mon[29133]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-19T03:11:57.512 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:57 smithi059 ceph-mon[29133]: osdmap e37: 6 total, 5 up, 6 in 2024-03-19T03:11:57.512 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:57 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:11:57.512 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:57 smithi059 ceph-mon[29133]: from='osd.5 [v2:172.21.15.176:6816/3279585260,v1:172.21.15.176:6817/3279585260]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-19T03:11:57.512 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:57 smithi059 ceph-mon[29133]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-19T03:11:57.512 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:57 smithi059 ceph-mon[29133]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:57.512 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:57 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:57.512 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:57 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:57.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:57 smithi176 ceph-mon[36450]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-19T03:11:57.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:57 smithi176 ceph-mon[36450]: osdmap e37: 6 total, 5 up, 6 in 2024-03-19T03:11:57.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:57 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:11:57.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:57 smithi176 ceph-mon[36450]: from='osd.5 [v2:172.21.15.176:6816/3279585260,v1:172.21.15.176:6817/3279585260]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-19T03:11:57.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:57 smithi176 ceph-mon[36450]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-19T03:11:57.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:57 smithi176 ceph-mon[36450]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-19T03:11:57.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:57 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:57.629 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:57 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:11:57.629 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 19 03:11:57 smithi176 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-5[55356]: 2024-03-19T03:11:57.268+0000 7f6813c83700 -1 osd.5 0 waiting for initial osdmap 2024-03-19T03:11:57.629 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 19 03:11:57 smithi176 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-5[55356]: 2024-03-19T03:11:57.275+0000 7f680ea7e700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T03:11:57.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.705+0000 7f8b3597a700 1 -- 172.21.15.59:0/2374917896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b300fffc0 msgr2=0x7f8b301023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:11:57.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.705+0000 7f8b3597a700 1 --2- 172.21.15.59:0/2374917896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b300fffc0 0x7f8b301023a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8b18004940 tx=0x7f8b18009e30 comp rx=0 tx=0).stop 2024-03-19T03:11:57.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.705+0000 7f8b3597a700 1 -- 172.21.15.59:0/2374917896 shutdown_connections 2024-03-19T03:11:57.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.705+0000 7f8b3597a700 1 --2- 172.21.15.59:0/2374917896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b300fffc0 0x7f8b301023a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:11:57.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.705+0000 7f8b3597a700 1 --2- 172.21.15.59:0/2374917896 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b30066e90 0x7f8b300ffa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:11:57.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.705+0000 7f8b3597a700 1 -- 172.21.15.59:0/2374917896 >> 172.21.15.59:0/2374917896 conn(0x7f8b300f7450 msgr2=0x7f8b300f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:11:57.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.705+0000 7f8b3597a700 1 -- 172.21.15.59:0/2374917896 shutdown_connections 2024-03-19T03:11:57.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.705+0000 7f8b3597a700 1 -- 172.21.15.59:0/2374917896 wait complete. 2024-03-19T03:11:57.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.706+0000 7f8b3597a700 1 Processor -- start 2024-03-19T03:11:57.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.706+0000 7f8b3597a700 1 -- start start 2024-03-19T03:11:57.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.706+0000 7f8b3597a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b30066e90 0x7f8b3010d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:11:57.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.706+0000 7f8b3597a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b300fffc0 0x7f8b3010de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:11:57.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.706+0000 7f8b3597a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3010e450 con 0x7f8b30066e90 2024-03-19T03:11:57.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.706+0000 7f8b3597a700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3010e5c0 con 0x7f8b300fffc0 2024-03-19T03:11:57.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.707+0000 7f8b2effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b30066e90 0x7f8b3010d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:11:57.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.707+0000 7f8b2effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b30066e90 0x7f8b3010d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47700/0 (socket says 172.21.15.59:47700) 2024-03-19T03:11:57.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.707+0000 7f8b2effd700 1 -- 172.21.15.59:0/721636089 learned_addr learned my addr 172.21.15.59:0/721636089 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:11:57.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.707+0000 7f8b2e7fc700 1 --2- 172.21.15.59:0/721636089 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b300fffc0 0x7f8b3010de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:11:57.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.707+0000 7f8b2effd700 1 -- 172.21.15.59:0/721636089 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b300fffc0 msgr2=0x7f8b3010de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:11:57.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.707+0000 7f8b2effd700 1 --2- 172.21.15.59:0/721636089 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b300fffc0 0x7f8b3010de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:11:57.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.707+0000 7f8b2effd700 1 -- 172.21.15.59:0/721636089 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b18005040 con 0x7f8b30066e90 2024-03-19T03:11:57.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.707+0000 7f8b2effd700 1 --2- 172.21.15.59:0/721636089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b30066e90 0x7f8b3010d910 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8b2000eb70 tx=0x7f8b2000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:11:57.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.707+0000 7f8b277fe700 1 -- 172.21.15.59:0/721636089 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b20007cc0 con 0x7f8b30066e90 2024-03-19T03:11:57.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.707+0000 7f8b3597a700 1 -- 172.21.15.59:0/721636089 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b30112fc0 con 0x7f8b30066e90 2024-03-19T03:11:57.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.707+0000 7f8b277fe700 1 -- 172.21.15.59:0/721636089 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b2000f040 con 0x7f8b30066e90 2024-03-19T03:11:57.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.707+0000 7f8b277fe700 1 -- 172.21.15.59:0/721636089 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b2000cb20 con 0x7f8b30066e90 2024-03-19T03:11:57.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.707+0000 7f8b3597a700 1 -- 172.21.15.59:0/721636089 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b30113470 con 0x7f8b30066e90 2024-03-19T03:11:57.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.708+0000 7f8b3597a700 1 -- 172.21.15.59:0/721636089 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b3004c710 con 0x7f8b30066e90 2024-03-19T03:11:57.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.709+0000 7f8b277fe700 1 -- 172.21.15.59:0/721636089 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f8b2000cc80 con 0x7f8b30066e90 2024-03-19T03:11:57.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.709+0000 7f8b277fe700 1 --2- 172.21.15.59:0/721636089 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f8b1c0692a0 0x7f8b1c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:11:57.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.709+0000 7f8b277fe700 1 -- 172.21.15.59:0/721636089 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4354+0+0 (secure 0 0 0) 0x7f8b200866a0 con 0x7f8b30066e90 2024-03-19T03:11:57.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.709+0000 7f8b2e7fc700 1 --2- 172.21.15.59:0/721636089 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f8b1c0692a0 0x7f8b1c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:11:57.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.709+0000 7f8b2e7fc700 1 --2- 172.21.15.59:0/721636089 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f8b1c0692a0 0x7f8b1c06b750 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8b18004890 tx=0x7f8b18009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:11:57.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.714+0000 7f8b277fe700 1 -- 172.21.15.59:0/721636089 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f8b2004de50 con 0x7f8b30066e90 2024-03-19T03:11:57.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.908+0000 7f8b3597a700 1 -- 172.21.15.59:0/721636089 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f8b30113a20 con 0x7f8b30066e90 2024-03-19T03:11:57.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.908+0000 7f8b277fe700 1 -- 172.21.15.59:0/721636089 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v38) v1 ==== 74+0+130 (secure 0 0 0) 0x7f8b20056960 con 0x7f8b30066e90 2024-03-19T03:11:57.910 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:11:57.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.910+0000 7f8b3597a700 1 -- 172.21.15.59:0/721636089 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f8b1c0692a0 msgr2=0x7f8b1c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:11:57.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.910+0000 7f8b3597a700 1 --2- 172.21.15.59:0/721636089 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f8b1c0692a0 0x7f8b1c06b750 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8b18004890 tx=0x7f8b18009e30 comp rx=0 tx=0).stop 2024-03-19T03:11:57.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.910+0000 7f8b3597a700 1 -- 172.21.15.59:0/721636089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b30066e90 msgr2=0x7f8b3010d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:11:57.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.910+0000 7f8b3597a700 1 --2- 172.21.15.59:0/721636089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b30066e90 0x7f8b3010d910 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8b2000eb70 tx=0x7f8b2000b040 comp rx=0 tx=0).stop 2024-03-19T03:11:57.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.910+0000 7f8b3597a700 1 -- 172.21.15.59:0/721636089 shutdown_connections 2024-03-19T03:11:57.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.910+0000 7f8b3597a700 1 --2- 172.21.15.59:0/721636089 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f8b1c0692a0 0x7f8b1c06b750 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:11:57.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.910+0000 7f8b3597a700 1 --2- 172.21.15.59:0/721636089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b30066e90 0x7f8b3010d910 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:11:57.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.910+0000 7f8b3597a700 1 --2- 172.21.15.59:0/721636089 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b300fffc0 0x7f8b3010de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:11:57.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.910+0000 7f8b3597a700 1 -- 172.21.15.59:0/721636089 >> 172.21.15.59:0/721636089 conn(0x7f8b300f7450 msgr2=0x7f8b300f9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:11:57.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.911+0000 7f8b3597a700 1 -- 172.21.15.59:0/721636089 shutdown_connections 2024-03-19T03:11:57.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:11:57.911+0000 7f8b3597a700 1 -- 172.21.15.59:0/721636089 wait complete. 2024-03-19T03:11:58.538 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:58 smithi176 ceph-mon[36450]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-03-19T03:11:58.539 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:58 smithi176 ceph-mon[36450]: osdmap e38: 6 total, 5 up, 6 in 2024-03-19T03:11:58.539 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:58 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:11:58.539 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:58 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:11:58.539 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:58 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/721636089' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-19T03:11:58.545 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:58 smithi059 ceph-mon[29133]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-03-19T03:11:58.545 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:58 smithi059 ceph-mon[29133]: osdmap e38: 6 total, 5 up, 6 in 2024-03-19T03:11:58.545 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:58 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:11:58.545 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:58 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:11:58.545 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:58 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/721636089' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-19T03:11:58.570 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":38,"num_osds":6,"num_up_osds":5,"osd_up_since":1710817893,"num_in_osds":6,"osd_in_since":1710817900,"num_remapped_pgs":0} 2024-03-19T03:11:59.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:59 smithi176 ceph-mon[36450]: purged_snaps scrub starts 2024-03-19T03:11:59.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:59 smithi176 ceph-mon[36450]: purged_snaps scrub ok 2024-03-19T03:11:59.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:59 smithi176 ceph-mon[36450]: osd.5 [v2:172.21.15.176:6816/3279585260,v1:172.21.15.176:6817/3279585260] boot 2024-03-19T03:11:59.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:59 smithi176 ceph-mon[36450]: osdmap e39: 6 total, 6 up, 6 in 2024-03-19T03:11:59.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:59 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:11:59.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:11:59 smithi176 ceph-mon[36450]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:11:59.572 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-19T03:11:59.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:59 smithi059 ceph-mon[29133]: purged_snaps scrub starts 2024-03-19T03:11:59.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:59 smithi059 ceph-mon[29133]: purged_snaps scrub ok 2024-03-19T03:11:59.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:59 smithi059 ceph-mon[29133]: osd.5 [v2:172.21.15.176:6816/3279585260,v1:172.21.15.176:6817/3279585260] boot 2024-03-19T03:11:59.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:59 smithi059 ceph-mon[29133]: osdmap e39: 6 total, 6 up, 6 in 2024-03-19T03:11:59.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:59 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:11:59.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:11:59 smithi059 ceph-mon[29133]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:11:59.835 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:00.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:00 smithi176 ceph-mon[36450]: osdmap e40: 6 total, 6 up, 6 in 2024-03-19T03:12:00.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:00 smithi059 ceph-mon[29133]: osdmap e40: 6 total, 6 up, 6 in 2024-03-19T03:12:01.648 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:01 smithi059 ceph-mon[29133]: Detected new or changed devices on smithi176 2024-03-19T03:12:01.648 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:12:01.648 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:12:01.648 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:12:01.648 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:12:01.648 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:12:01.648 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:01 smithi059 ceph-mon[29133]: Adjusting osd_memory_target on smithi176 to 5312M 2024-03-19T03:12:01.648 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:12:01.648 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:12:01.648 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:12:01.648 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:12:01.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:12:01.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:12:01.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:12:01.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:12:01.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:01 smithi059 ceph-mon[29133]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:01.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:12:01.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.678+0000 7fab2d9cf700 1 -- 172.21.15.59:0/3880259653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab280fbcd0 msgr2=0x7fab280fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:01.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.678+0000 7fab2d9cf700 1 --2- 172.21.15.59:0/3880259653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab280fbcd0 0x7fab280fc0e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fab18004940 tx=0x7fab18009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:01.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.678+0000 7fab2d9cf700 1 -- 172.21.15.59:0/3880259653 shutdown_connections 2024-03-19T03:12:01.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.678+0000 7fab2d9cf700 1 --2- 172.21.15.59:0/3880259653 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab280fcf80 0x7fab280fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:01.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.678+0000 7fab2d9cf700 1 --2- 172.21.15.59:0/3880259653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab280fbcd0 0x7fab280fc0e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:01.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.678+0000 7fab2d9cf700 1 -- 172.21.15.59:0/3880259653 >> 172.21.15.59:0/3880259653 conn(0x7fab280f7450 msgr2=0x7fab280f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:01.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.678+0000 7fab2d9cf700 1 -- 172.21.15.59:0/3880259653 shutdown_connections 2024-03-19T03:12:01.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.678+0000 7fab2d9cf700 1 -- 172.21.15.59:0/3880259653 wait complete. 2024-03-19T03:12:01.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.679+0000 7fab2d9cf700 1 Processor -- start 2024-03-19T03:12:01.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.679+0000 7fab2d9cf700 1 -- start start 2024-03-19T03:12:01.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.679+0000 7fab2d9cf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab280fbcd0 0x7fab280ffbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:01.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.679+0000 7fab2d9cf700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab280fcf80 0x7fab280fe2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:01.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.679+0000 7fab2d9cf700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab281001c0 con 0x7fab280fbcd0 2024-03-19T03:12:01.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.679+0000 7fab2d9cf700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab280fe800 con 0x7fab280fcf80 2024-03-19T03:12:01.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.679+0000 7fab26ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab280fbcd0 0x7fab280ffbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:01.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.679+0000 7fab26ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab280fbcd0 0x7fab280ffbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47728/0 (socket says 172.21.15.59:47728) 2024-03-19T03:12:01.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.679+0000 7fab26ffd700 1 -- 172.21.15.59:0/559656261 learned_addr learned my addr 172.21.15.59:0/559656261 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:01.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.680+0000 7fab267fc700 1 --2- 172.21.15.59:0/559656261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab280fcf80 0x7fab280fe2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:01.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.680+0000 7fab26ffd700 1 -- 172.21.15.59:0/559656261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab280fcf80 msgr2=0x7fab280fe2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:01.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.680+0000 7fab26ffd700 1 --2- 172.21.15.59:0/559656261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab280fcf80 0x7fab280fe2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:01.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.680+0000 7fab26ffd700 1 -- 172.21.15.59:0/559656261 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab18005040 con 0x7fab280fbcd0 2024-03-19T03:12:01.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.680+0000 7fab26ffd700 1 --2- 172.21.15.59:0/559656261 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab280fbcd0 0x7fab280ffbc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fab1800b3b0 tx=0x7fab18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:01.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.680+0000 7fab0f7fe700 1 -- 172.21.15.59:0/559656261 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab1800e040 con 0x7fab280fbcd0 2024-03-19T03:12:01.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.680+0000 7fab0f7fe700 1 -- 172.21.15.59:0/559656261 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab1800bbf0 con 0x7fab280fbcd0 2024-03-19T03:12:01.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.680+0000 7fab2d9cf700 1 -- 172.21.15.59:0/559656261 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab280fea80 con 0x7fab280fbcd0 2024-03-19T03:12:01.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.680+0000 7fab0f7fe700 1 -- 172.21.15.59:0/559656261 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab1801c630 con 0x7fab280fbcd0 2024-03-19T03:12:01.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.680+0000 7fab2d9cf700 1 -- 172.21.15.59:0/559656261 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab280fefc0 con 0x7fab280fbcd0 2024-03-19T03:12:01.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.681+0000 7fab2d9cf700 1 -- 172.21.15.59:0/559656261 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab28101b10 con 0x7fab280fbcd0 2024-03-19T03:12:01.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.682+0000 7fab0f7fe700 1 -- 172.21.15.59:0/559656261 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fab1800bd60 con 0x7fab280fbcd0 2024-03-19T03:12:01.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.682+0000 7fab0f7fe700 1 --2- 172.21.15.59:0/559656261 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fab100692a0 0x7fab1006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:01.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.682+0000 7fab267fc700 1 --2- 172.21.15.59:0/559656261 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fab100692a0 0x7fab1006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:01.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.682+0000 7fab0f7fe700 1 -- 172.21.15.59:0/559656261 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fab18087a50 con 0x7fab280fbcd0 2024-03-19T03:12:01.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.683+0000 7fab267fc700 1 --2- 172.21.15.59:0/559656261 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fab100692a0 0x7fab1006b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fab1c004f40 tx=0x7fab1c00b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:01.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.687+0000 7fab0f7fe700 1 -- 172.21.15.59:0/559656261 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fab18057bf0 con 0x7fab280fbcd0 2024-03-19T03:12:01.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:01 smithi176 ceph-mon[36450]: Detected new or changed devices on smithi176 2024-03-19T03:12:01.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:12:01.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:12:01.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:12:01.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:12:01.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:12:01.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:01 smithi176 ceph-mon[36450]: Adjusting osd_memory_target on smithi176 to 5312M 2024-03-19T03:12:01.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:12:01.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:12:01.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:12:01.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:12:01.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:12:01.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:12:01.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:12:01.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:12:01.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:01 smithi176 ceph-mon[36450]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:01.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:12:01.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.880+0000 7fab2d9cf700 1 -- 172.21.15.59:0/559656261 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fab2804c710 con 0x7fab280fbcd0 2024-03-19T03:12:01.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.880+0000 7fab0f7fe700 1 -- 172.21.15.59:0/559656261 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7fab2804c710 con 0x7fab280fbcd0 2024-03-19T03:12:01.882 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:12:01.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.882+0000 7fab2d9cf700 1 -- 172.21.15.59:0/559656261 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fab100692a0 msgr2=0x7fab1006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:01.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.882+0000 7fab2d9cf700 1 --2- 172.21.15.59:0/559656261 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fab100692a0 0x7fab1006b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fab1c004f40 tx=0x7fab1c00b040 comp rx=0 tx=0).stop 2024-03-19T03:12:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.882+0000 7fab2d9cf700 1 -- 172.21.15.59:0/559656261 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab280fbcd0 msgr2=0x7fab280ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.882+0000 7fab2d9cf700 1 --2- 172.21.15.59:0/559656261 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab280fbcd0 0x7fab280ffbc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fab1800b3b0 tx=0x7fab18009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.882+0000 7fab2d9cf700 1 -- 172.21.15.59:0/559656261 shutdown_connections 2024-03-19T03:12:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.882+0000 7fab2d9cf700 1 --2- 172.21.15.59:0/559656261 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fab100692a0 0x7fab1006b750 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.882+0000 7fab2d9cf700 1 --2- 172.21.15.59:0/559656261 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab280fbcd0 0x7fab280ffbc0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.882+0000 7fab2d9cf700 1 --2- 172.21.15.59:0/559656261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab280fcf80 0x7fab280fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.882+0000 7fab2d9cf700 1 -- 172.21.15.59:0/559656261 >> 172.21.15.59:0/559656261 conn(0x7fab280f7450 msgr2=0x7fab28066870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.882+0000 7fab2d9cf700 1 -- 172.21.15.59:0/559656261 shutdown_connections 2024-03-19T03:12:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:01.882+0000 7fab2d9cf700 1 -- 172.21.15.59:0/559656261 wait complete. 2024-03-19T03:12:02.570 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1710817918,"num_in_osds":6,"osd_in_since":1710817900,"num_remapped_pgs":0} 2024-03-19T03:12:02.570 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-19T03:12:02.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:02 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/559656261' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-19T03:12:02.874 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:02.938 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:02 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/559656261' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-19T03:12:03.990 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:03 smithi059 ceph-mon[29133]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:04.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:03 smithi176 ceph-mon[36450]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:04.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.466+0000 7f499f81e700 1 -- 172.21.15.59:0/3097676766 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49980fb540 msgr2=0x7f49980fb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:04.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.466+0000 7f499f81e700 1 --2- 172.21.15.59:0/3097676766 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49980fb540 0x7f49980fb990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f498c004940 tx=0x7f498c009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:04.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.466+0000 7f499f81e700 1 -- 172.21.15.59:0/3097676766 shutdown_connections 2024-03-19T03:12:04.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.466+0000 7f499f81e700 1 --2- 172.21.15.59:0/3097676766 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49980fb540 0x7f49980fb990 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:04.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.466+0000 7f499f81e700 1 --2- 172.21.15.59:0/3097676766 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4998102fd0 0x7f49981033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:04.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.466+0000 7f499f81e700 1 -- 172.21.15.59:0/3097676766 >> 172.21.15.59:0/3097676766 conn(0x7f4998071530 msgr2=0x7f4998071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:04.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.466+0000 7f499f81e700 1 -- 172.21.15.59:0/3097676766 shutdown_connections 2024-03-19T03:12:04.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.466+0000 7f499f81e700 1 -- 172.21.15.59:0/3097676766 wait complete. 2024-03-19T03:12:04.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.467+0000 7f499f81e700 1 Processor -- start 2024-03-19T03:12:04.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.467+0000 7f499f81e700 1 -- start start 2024-03-19T03:12:04.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.467+0000 7f499f81e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49980fb540 0x7f4998113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:04.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.467+0000 7f499f81e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4998102fd0 0x7f4998113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:04.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.467+0000 7f499f81e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49981145b0 con 0x7f49980fb540 2024-03-19T03:12:04.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.467+0000 7f499f81e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499810db40 con 0x7f4998102fd0 2024-03-19T03:12:04.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.467+0000 7f499d5ba700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49980fb540 0x7f4998113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:04.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.467+0000 7f499d5ba700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49980fb540 0x7f4998113990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36316/0 (socket says 172.21.15.59:36316) 2024-03-19T03:12:04.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.467+0000 7f499d5ba700 1 -- 172.21.15.59:0/1636886520 learned_addr learned my addr 172.21.15.59:0/1636886520 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:04.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.467+0000 7f499cdb9700 1 --2- 172.21.15.59:0/1636886520 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4998102fd0 0x7f4998113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:04.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.467+0000 7f499d5ba700 1 -- 172.21.15.59:0/1636886520 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4998102fd0 msgr2=0x7f4998113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:04.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.467+0000 7f499d5ba700 1 --2- 172.21.15.59:0/1636886520 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4998102fd0 0x7f4998113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:04.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.467+0000 7f499d5ba700 1 -- 172.21.15.59:0/1636886520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498c005040 con 0x7f49980fb540 2024-03-19T03:12:04.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.467+0000 7f499d5ba700 1 --2- 172.21.15.59:0/1636886520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49980fb540 0x7f4998113990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f499000b790 tx=0x7f4990009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:04.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.468+0000 7f4995ffb700 1 -- 172.21.15.59:0/1636886520 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f499000f770 con 0x7f49980fb540 2024-03-19T03:12:04.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.468+0000 7f4995ffb700 1 -- 172.21.15.59:0/1636886520 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f499000fda0 con 0x7f49980fb540 2024-03-19T03:12:04.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.468+0000 7f499f81e700 1 -- 172.21.15.59:0/1636886520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f499810de20 con 0x7f49980fb540 2024-03-19T03:12:04.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.468+0000 7f4995ffb700 1 -- 172.21.15.59:0/1636886520 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f499000d310 con 0x7f49980fb540 2024-03-19T03:12:04.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.468+0000 7f499f81e700 1 -- 172.21.15.59:0/1636886520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f499810e360 con 0x7f49980fb540 2024-03-19T03:12:04.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.469+0000 7f499f81e700 1 -- 172.21.15.59:0/1636886520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4998100270 con 0x7f49980fb540 2024-03-19T03:12:04.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.469+0000 7f4995ffb700 1 -- 172.21.15.59:0/1636886520 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4990010420 con 0x7f49980fb540 2024-03-19T03:12:04.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.469+0000 7f4995ffb700 1 --2- 172.21.15.59:0/1636886520 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f49800692a0 0x7f498006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:04.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.469+0000 7f499cdb9700 1 --2- 172.21.15.59:0/1636886520 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f49800692a0 0x7f498006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:04.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.470+0000 7f4995ffb700 1 -- 172.21.15.59:0/1636886520 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4990086e70 con 0x7f49980fb540 2024-03-19T03:12:04.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.470+0000 7f499cdb9700 1 --2- 172.21.15.59:0/1636886520 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f49800692a0 0x7f498006b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f498c000c00 tx=0x7f498c009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:04.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.474+0000 7f4995ffb700 1 -- 172.21.15.59:0/1636886520 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f49900534e0 con 0x7f49980fb540 2024-03-19T03:12:04.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.665+0000 7f499f81e700 1 -- 172.21.15.59:0/1636886520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49980fe860 con 0x7f49980fb540 2024-03-19T03:12:04.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.666+0000 7f4995ffb700 1 -- 172.21.15.59:0/1636886520 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11081 (secure 0 0 0) 0x7f4990056b00 con 0x7f49980fb540 2024-03-19T03:12:04.668 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:12:04.668 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":40,"fsid":"8c8ae530-e59d-11ee-95c9-87774f69a715","created":"2024-03-19T03:06:15.064131+0000","modified":"2024-03-19T03:11:59.268254+0000","last_up_change":"2024-03-19T03:11:58.260172+0000","last_in_change":"2024-03-19T03:11:40.023566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:10:45.238939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"f6a12d03-3dd9-40d4-b54e-4b2041c18e48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1770897334},{"type":"v1","addr":"172.21.15.59:6803","nonce":1770897334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1770897334},{"type":"v1","addr":"172.21.15.59:6805","nonce":1770897334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1770897334},{"type":"v1","addr":"172.21.15.59:6809","nonce":1770897334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1770897334},{"type":"v1","addr":"172.21.15.59:6807","nonce":1770897334}]},"public_addr":"172.21.15.59:6803/1770897334","cluster_addr":"172.21.15.59:6805/1770897334","heartbeat_back_addr":"172.21.15.59:6809/1770897334","heartbeat_front_addr":"172.21.15.59:6807/1770897334","state":["exists","up"]},{"osd":1,"uuid":"c1069548-eb23-476b-a661-b71f0a73987e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3595398218},{"type":"v1","addr":"172.21.15.59:6811","nonce":3595398218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3595398218},{"type":"v1","addr":"172.21.15.59:6813","nonce":3595398218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":3595398218},{"type":"v1","addr":"172.21.15.59:6817","nonce":3595398218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3595398218},{"type":"v1","addr":"172.21.15.59:6815","nonce":3595398218}]},"public_addr":"172.21.15.59:6811/3595398218","cluster_addr":"172.21.15.59:6813/3595398218","heartbeat_back_addr":"172.21.15.59:6817/3595398218","heartbeat_front_addr":"172.21.15.59:6815/3595398218","state":["exists","up"]},{"osd":2,"uuid":"4958f1e4-2dc9-458f-82ff-e51d98762cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":639205676},{"type":"v1","addr":"172.21.15.59:6819","nonce":639205676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":639205676},{"type":"v1","addr":"172.21.15.59:6821","nonce":639205676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":639205676},{"type":"v1","addr":"172.21.15.59:6825","nonce":639205676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":639205676},{"type":"v1","addr":"172.21.15.59:6823","nonce":639205676}]},"public_addr":"172.21.15.59:6819/639205676","cluster_addr":"172.21.15.59:6821/639205676","heartbeat_back_addr":"172.21.15.59:6825/639205676","heartbeat_front_addr":"172.21.15.59:6823/639205676","state":["exists","up"]},{"osd":3,"uuid":"c3cf9095-7ce2-4d0f-bc65-2cda09504d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":3291292674},{"type":"v1","addr":"172.21.15.176:6801","nonce":3291292674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":3291292674},{"type":"v1","addr":"172.21.15.176:6803","nonce":3291292674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":3291292674},{"type":"v1","addr":"172.21.15.176:6807","nonce":3291292674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":3291292674},{"type":"v1","addr":"172.21.15.176:6805","nonce":3291292674}]},"public_addr":"172.21.15.176:6801/3291292674","cluster_addr":"172.21.15.176:6803/3291292674","heartbeat_back_addr":"172.21.15.176:6807/3291292674","heartbeat_front_addr":"172.21.15.176:6805/3291292674","state":["exists","up"]},{"osd":4,"uuid":"b102d232-9f65-4d96-ad81-b3d1a0b8faf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":1006246702},{"type":"v1","addr":"172.21.15.176:6809","nonce":1006246702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":1006246702},{"type":"v1","addr":"172.21.15.176:6811","nonce":1006246702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":1006246702},{"type":"v1","addr":"172.21.15.176:6815","nonce":1006246702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":1006246702},{"type":"v1","addr":"172.21.15.176:6813","nonce":1006246702}]},"public_addr":"172.21.15.176:6809/1006246702","cluster_addr":"172.21.15.176:6811/1006246702","heartbeat_back_addr":"172.21.15.176:6815/1006246702","heartbeat_front_addr":"172.21.15.176:6813/1006246702","state":["exists","up"]},{"osd":5,"uuid":"f9c805ed-8aa0-4f39-a78f-c6a4b018eb7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":3279585260},{"type":"v1","addr":"172.21.15.176:6817","nonce":3279585260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":3279585260},{"type":"v1","addr":"172.21.15.176:6819","nonce":3279585260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":3279585260},{"type":"v1","addr":"172.21.15.176:6823","nonce":3279585260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":3279585260},{"type":"v1","addr":"172.21.15.176:6821","nonce":3279585260}]},"public_addr":"172.21.15.176:6817/3279585260","cluster_addr":"172.21.15.176:6819/3279585260","heartbeat_back_addr":"172.21.15.176:6823/3279585260","heartbeat_front_addr":"172.21.15.176:6821/3279585260","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:09:44.576635+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:10:12.913840+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:10:42.022990+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:11:05.039788+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:11:31.198904+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:11:57.135656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:0/2117971082":"2024-03-20T03:08:30.903133+0000","172.21.15.59:0/988882608":"2024-03-20T03:07:20.738216+0000","172.21.15.59:0/2884557489":"2024-03-20T03:08:30.903133+0000","172.21.15.59:6801/1573642037":"2024-03-20T03:07:20.738216+0000","172.21.15.59:0/2956286237":"2024-03-20T03:07:20.738216+0000","172.21.15.59:6801/1857243897":"2024-03-20T03:08:30.903133+0000","172.21.15.59:0/661762888":"2024-03-20T03:06:49.795873+0000","172.21.15.59:0/2659339884":"2024-03-20T03:06:49.795873+0000","172.21.15.59:0/156375080":"2024-03-20T03:06:49.795873+0000","172.21.15.59:6800/3609017287":"2024-03-20T03:06:49.795873+0000","172.21.15.59:0/133172490":"2024-03-20T03:08:30.903133+0000","172.21.15.59:6800/1573642037":"2024-03-20T03:07:20.738216+0000","172.21.15.59:6801/3609017287":"2024-03-20T03:06:49.795873+0000","172.21.15.59:6800/1857243897":"2024-03-20T03:08:30.903133+0000","172.21.15.59:0/2483030805":"2024-03-20T03:07:20.738216+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:12:04.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.668+0000 7f499f81e700 1 -- 172.21.15.59:0/1636886520 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f49800692a0 msgr2=0x7f498006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:04.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.668+0000 7f499f81e700 1 --2- 172.21.15.59:0/1636886520 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f49800692a0 0x7f498006b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f498c000c00 tx=0x7f498c009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:04.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.668+0000 7f499f81e700 1 -- 172.21.15.59:0/1636886520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49980fb540 msgr2=0x7f4998113990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:04.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.668+0000 7f499f81e700 1 --2- 172.21.15.59:0/1636886520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49980fb540 0x7f4998113990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f499000b790 tx=0x7f4990009250 comp rx=0 tx=0).stop 2024-03-19T03:12:04.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.668+0000 7f499f81e700 1 -- 172.21.15.59:0/1636886520 shutdown_connections 2024-03-19T03:12:04.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.668+0000 7f499f81e700 1 --2- 172.21.15.59:0/1636886520 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f49800692a0 0x7f498006b750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:04.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.668+0000 7f499f81e700 1 --2- 172.21.15.59:0/1636886520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49980fb540 0x7f4998113990 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:04.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.668+0000 7f499f81e700 1 --2- 172.21.15.59:0/1636886520 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4998102fd0 0x7f4998113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:04.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.668+0000 7f499f81e700 1 -- 172.21.15.59:0/1636886520 >> 172.21.15.59:0/1636886520 conn(0x7f4998071530 msgr2=0x7f49980fa580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:04.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.668+0000 7f499f81e700 1 -- 172.21.15.59:0/1636886520 shutdown_connections 2024-03-19T03:12:04.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:04.668+0000 7f499f81e700 1 -- 172.21.15.59:0/1636886520 wait complete. 2024-03-19T03:12:05.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:04 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/1636886520' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:12:05.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:04 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/1636886520' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:12:05.381 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T03:10:45.238939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-03-19T03:12:05.381 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-19T03:12:05.641 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:05.944 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:05 smithi059 ceph-mon[29133]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:06.030 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:05 smithi176 ceph-mon[36450]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:07.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.140+0000 7facaf972700 1 -- 172.21.15.59:0/2257588694 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faca8102710 msgr2=0x7faca80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:07.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.140+0000 7facaf972700 1 --2- 172.21.15.59:0/2257588694 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faca8102710 0x7faca80fbd80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fac98004940 tx=0x7fac98009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:07.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.140+0000 7facaf972700 1 -- 172.21.15.59:0/2257588694 shutdown_connections 2024-03-19T03:12:07.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.140+0000 7facaf972700 1 --2- 172.21.15.59:0/2257588694 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faca8102710 0x7faca80fbd80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:07.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.140+0000 7facaf972700 1 --2- 172.21.15.59:0/2257588694 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faca8101d70 0x7faca8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:07.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.140+0000 7facaf972700 1 -- 172.21.15.59:0/2257588694 >> 172.21.15.59:0/2257588694 conn(0x7faca8072bc0 msgr2=0x7faca8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:07.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.140+0000 7facaf972700 1 -- 172.21.15.59:0/2257588694 shutdown_connections 2024-03-19T03:12:07.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.140+0000 7facaf972700 1 -- 172.21.15.59:0/2257588694 wait complete. 2024-03-19T03:12:07.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.141+0000 7facaf972700 1 Processor -- start 2024-03-19T03:12:07.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.141+0000 7facaf972700 1 -- start start 2024-03-19T03:12:07.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.141+0000 7facaf972700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faca8101d70 0x7faca8113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:07.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.141+0000 7facaf972700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faca8102710 0x7faca81140d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:07.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.141+0000 7facaf972700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca8114720 con 0x7faca8101d70 2024-03-19T03:12:07.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.141+0000 7facaf972700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca810db10 con 0x7faca8102710 2024-03-19T03:12:07.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.141+0000 7facad70e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faca8101d70 0x7faca8113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:07.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.141+0000 7facad70e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faca8101d70 0x7faca8113b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36336/0 (socket says 172.21.15.59:36336) 2024-03-19T03:12:07.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.141+0000 7facad70e700 1 -- 172.21.15.59:0/251310014 learned_addr learned my addr 172.21.15.59:0/251310014 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:07.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.141+0000 7facacf0d700 1 --2- 172.21.15.59:0/251310014 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faca8102710 0x7faca81140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:07.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.141+0000 7facad70e700 1 -- 172.21.15.59:0/251310014 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faca8102710 msgr2=0x7faca81140d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:07.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.141+0000 7facad70e700 1 --2- 172.21.15.59:0/251310014 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faca8102710 0x7faca81140d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:07.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.141+0000 7facad70e700 1 -- 172.21.15.59:0/251310014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac98005040 con 0x7faca8101d70 2024-03-19T03:12:07.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.142+0000 7facad70e700 1 --2- 172.21.15.59:0/251310014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faca8101d70 0x7faca8113b90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faca400c8c0 tx=0x7faca40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:07.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.142+0000 7fac9dffb700 1 -- 172.21.15.59:0/251310014 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faca400a760 con 0x7faca8101d70 2024-03-19T03:12:07.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.142+0000 7facaf972700 1 -- 172.21.15.59:0/251310014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faca810ddf0 con 0x7faca8101d70 2024-03-19T03:12:07.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.142+0000 7fac9dffb700 1 -- 172.21.15.59:0/251310014 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faca400ad90 con 0x7faca8101d70 2024-03-19T03:12:07.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.142+0000 7fac9dffb700 1 -- 172.21.15.59:0/251310014 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faca40113a0 con 0x7faca8101d70 2024-03-19T03:12:07.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.142+0000 7facaf972700 1 -- 172.21.15.59:0/251310014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faca810e380 con 0x7faca8101d70 2024-03-19T03:12:07.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.143+0000 7facaf972700 1 -- 172.21.15.59:0/251310014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faca80fd4c0 con 0x7faca8101d70 2024-03-19T03:12:07.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.143+0000 7fac9dffb700 1 -- 172.21.15.59:0/251310014 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7faca4012400 con 0x7faca8101d70 2024-03-19T03:12:07.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.143+0000 7fac9dffb700 1 --2- 172.21.15.59:0/251310014 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fac940692a0 0x7fac9406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:07.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.144+0000 7fac9dffb700 1 -- 172.21.15.59:0/251310014 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7faca4086e60 con 0x7faca8101d70 2024-03-19T03:12:07.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.144+0000 7facacf0d700 1 --2- 172.21.15.59:0/251310014 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fac940692a0 0x7fac9406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:07.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.144+0000 7facacf0d700 1 --2- 172.21.15.59:0/251310014 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fac940692a0 0x7fac9406b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faca810edf0 tx=0x7fac98009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:07.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.148+0000 7fac9dffb700 1 -- 172.21.15.59:0/251310014 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7faca4056f60 con 0x7faca8101d70 2024-03-19T03:12:07.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.340+0000 7facaf972700 1 -- 172.21.15.59:0/251310014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faca8108e50 con 0x7faca8101d70 2024-03-19T03:12:07.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.340+0000 7fac9dffb700 1 -- 172.21.15.59:0/251310014 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7faca4056af0 con 0x7faca8101d70 2024-03-19T03:12:07.342 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-19T03:12:07.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.342+0000 7facaf972700 1 -- 172.21.15.59:0/251310014 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fac940692a0 msgr2=0x7fac9406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:07.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.342+0000 7facaf972700 1 --2- 172.21.15.59:0/251310014 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fac940692a0 0x7fac9406b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faca810edf0 tx=0x7fac98009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:07.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.342+0000 7facaf972700 1 -- 172.21.15.59:0/251310014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faca8101d70 msgr2=0x7faca8113b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:07.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.342+0000 7facaf972700 1 --2- 172.21.15.59:0/251310014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faca8101d70 0x7faca8113b90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faca400c8c0 tx=0x7faca40044c0 comp rx=0 tx=0).stop 2024-03-19T03:12:07.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.342+0000 7facaf972700 1 -- 172.21.15.59:0/251310014 shutdown_connections 2024-03-19T03:12:07.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.342+0000 7facaf972700 1 --2- 172.21.15.59:0/251310014 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fac940692a0 0x7fac9406b750 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:07.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.342+0000 7facaf972700 1 --2- 172.21.15.59:0/251310014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faca8101d70 0x7faca8113b90 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:07.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.342+0000 7facaf972700 1 --2- 172.21.15.59:0/251310014 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faca8102710 0x7faca81140d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:07.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.342+0000 7facaf972700 1 -- 172.21.15.59:0/251310014 >> 172.21.15.59:0/251310014 conn(0x7faca8072bc0 msgr2=0x7faca8100590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:07.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.342+0000 7facaf972700 1 -- 172.21.15.59:0/251310014 shutdown_connections 2024-03-19T03:12:07.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:07.342+0000 7facaf972700 1 -- 172.21.15.59:0/251310014 wait complete. 2024-03-19T03:12:08.032 INFO:tasks.cephadm:Setting up client nodes... 2024-03-19T03:12:08.032 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-19T03:12:08.296 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:08.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:08 smithi059 ceph-mon[29133]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:08.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:08 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/251310014' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:12:08.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:08 smithi176 ceph-mon[36450]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:08.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:08 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/251310014' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:12:09.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.954+0000 7f69a11a0700 1 -- 172.21.15.59:0/1204613631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f699c0fcdc0 msgr2=0x7f699c0fd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:09.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.954+0000 7f69a11a0700 1 --2- 172.21.15.59:0/1204613631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f699c0fcdc0 0x7f699c0fd230 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6990004940 tx=0x7f6990009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:09.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.954+0000 7f69a11a0700 1 -- 172.21.15.59:0/1204613631 shutdown_connections 2024-03-19T03:12:09.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.954+0000 7f69a11a0700 1 --2- 172.21.15.59:0/1204613631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f699c0fcdc0 0x7f699c0fd230 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:09.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.954+0000 7f69a11a0700 1 --2- 172.21.15.59:0/1204613631 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f699c0fbb10 0x7f699c0fbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:09.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.954+0000 7f69a11a0700 1 -- 172.21.15.59:0/1204613631 >> 172.21.15.59:0/1204613631 conn(0x7f699c0f7270 msgr2=0x7f699c0f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:09.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.954+0000 7f69a11a0700 1 -- 172.21.15.59:0/1204613631 shutdown_connections 2024-03-19T03:12:09.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.954+0000 7f69a11a0700 1 -- 172.21.15.59:0/1204613631 wait complete. 2024-03-19T03:12:09.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.955+0000 7f69a11a0700 1 Processor -- start 2024-03-19T03:12:09.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.955+0000 7f69a11a0700 1 -- start start 2024-03-19T03:12:09.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.955+0000 7f69a11a0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f699c0fbb10 0x7f699c10faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:09.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.955+0000 7f69a11a0700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f699c0fcdc0 0x7f699c110030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:09.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.955+0000 7f69a11a0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c110630 con 0x7f699c0fbb10 2024-03-19T03:12:09.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.955+0000 7f69a11a0700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c110e30 con 0x7f699c0fcdc0 2024-03-19T03:12:09.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.955+0000 7f699bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f699c0fbb10 0x7f699c10faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:09.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.955+0000 7f699bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f699c0fbb10 0x7f699c10faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36350/0 (socket says 172.21.15.59:36350) 2024-03-19T03:12:09.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.955+0000 7f699bfff700 1 -- 172.21.15.59:0/2708134128 learned_addr learned my addr 172.21.15.59:0/2708134128 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:09.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.955+0000 7f699b7fe700 1 --2- 172.21.15.59:0/2708134128 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f699c0fcdc0 0x7f699c110030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:09.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.955+0000 7f699bfff700 1 -- 172.21.15.59:0/2708134128 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f699c0fcdc0 msgr2=0x7f699c110030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:09.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.955+0000 7f699bfff700 1 --2- 172.21.15.59:0/2708134128 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f699c0fcdc0 0x7f699c110030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:09.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.955+0000 7f699bfff700 1 -- 172.21.15.59:0/2708134128 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6990005040 con 0x7f699c0fbb10 2024-03-19T03:12:09.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.956+0000 7f699bfff700 1 --2- 172.21.15.59:0/2708134128 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f699c0fbb10 0x7f699c10faf0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f698c002a50 tx=0x7f698c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:09.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.956+0000 7f6998ff9700 1 -- 172.21.15.59:0/2708134128 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f698c007cc0 con 0x7f699c0fbb10 2024-03-19T03:12:09.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.956+0000 7f69a11a0700 1 -- 172.21.15.59:0/2708134128 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f699c111110 con 0x7f699c0fbb10 2024-03-19T03:12:09.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.956+0000 7f6998ff9700 1 -- 172.21.15.59:0/2708134128 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f698c00f040 con 0x7f699c0fbb10 2024-03-19T03:12:09.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.956+0000 7f6998ff9700 1 -- 172.21.15.59:0/2708134128 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f698c00da30 con 0x7f699c0fbb10 2024-03-19T03:12:09.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.956+0000 7f69a11a0700 1 -- 172.21.15.59:0/2708134128 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f699c111650 con 0x7f699c0fbb10 2024-03-19T03:12:09.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.957+0000 7f69a11a0700 1 -- 172.21.15.59:0/2708134128 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f699c101750 con 0x7f699c0fbb10 2024-03-19T03:12:09.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.957+0000 7f6998ff9700 1 -- 172.21.15.59:0/2708134128 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f698c00dbb0 con 0x7f699c0fbb10 2024-03-19T03:12:09.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.957+0000 7f6998ff9700 1 --2- 172.21.15.59:0/2708134128 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f69840692a0 0x7f698406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:09.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.958+0000 7f6998ff9700 1 -- 172.21.15.59:0/2708134128 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f698c086be0 con 0x7f699c0fbb10 2024-03-19T03:12:09.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.958+0000 7f699b7fe700 1 --2- 172.21.15.59:0/2708134128 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f69840692a0 0x7f698406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:09.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.958+0000 7f699b7fe700 1 --2- 172.21.15.59:0/2708134128 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f69840692a0 0x7f698406b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6990009fd0 tx=0x7f6990009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:09.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:09.962+0000 7f6998ff9700 1 -- 172.21.15.59:0/2708134128 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f698c056c60 con 0x7f699c0fbb10 2024-03-19T03:12:10.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:10.229+0000 7f69a11a0700 1 -- 172.21.15.59:0/2708134128 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f699c0faf10 con 0x7f699c0fbb10 2024-03-19T03:12:10.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:10.233+0000 7f6998ff9700 1 -- 172.21.15.59:0/2708134128 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f698c0567f0 con 0x7f699c0fbb10 2024-03-19T03:12:10.235 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-03-19T03:12:10.235 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCKAvllacbEDRAACuV0mflEyYAeCEeAdWnthw== 2024-03-19T03:12:10.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:10.235+0000 7f69a11a0700 1 -- 172.21.15.59:0/2708134128 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f69840692a0 msgr2=0x7f698406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:10.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:10.235+0000 7f69a11a0700 1 --2- 172.21.15.59:0/2708134128 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f69840692a0 0x7f698406b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6990009fd0 tx=0x7f6990009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:10.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:10.235+0000 7f69a11a0700 1 -- 172.21.15.59:0/2708134128 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f699c0fbb10 msgr2=0x7f699c10faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:10.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:10.235+0000 7f69a11a0700 1 --2- 172.21.15.59:0/2708134128 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f699c0fbb10 0x7f699c10faf0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f698c002a50 tx=0x7f698c0046b0 comp rx=0 tx=0).stop 2024-03-19T03:12:10.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:10.235+0000 7f69a11a0700 1 -- 172.21.15.59:0/2708134128 shutdown_connections 2024-03-19T03:12:10.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:10.235+0000 7f69a11a0700 1 --2- 172.21.15.59:0/2708134128 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f69840692a0 0x7f698406b750 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:10.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:10.235+0000 7f69a11a0700 1 --2- 172.21.15.59:0/2708134128 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f699c0fbb10 0x7f699c10faf0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:10.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:10.235+0000 7f69a11a0700 1 --2- 172.21.15.59:0/2708134128 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f699c0fcdc0 0x7f699c110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:10.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:10.235+0000 7f69a11a0700 1 -- 172.21.15.59:0/2708134128 >> 172.21.15.59:0/2708134128 conn(0x7f699c0f7270 msgr2=0x7f699c065b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:10.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:10.235+0000 7f69a11a0700 1 -- 172.21.15.59:0/2708134128 shutdown_connections 2024-03-19T03:12:10.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:10.235+0000 7f69a11a0700 1 -- 172.21.15.59:0/2708134128 wait complete. 2024-03-19T03:12:10.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:10 smithi059 ceph-mon[29133]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:10.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:10 smithi176 ceph-mon[36450]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:10.807 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-19T03:12:10.807 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-19T03:12:10.808 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-19T03:12:10.850 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-19T03:12:11.115 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi176/config 2024-03-19T03:12:11.237 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:11 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/2708134128' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T03:12:11.237 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:11 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/2708134128' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-19T03:12:11.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:11 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/2708134128' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T03:12:11.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:11 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/2708134128' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-19T03:12:12.270 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:12 smithi176 ceph-mon[36450]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:12.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:12 smithi059 ceph-mon[29133]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:12.765 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.762+0000 7fc94d3fa700 1 -- 172.21.15.176:0/3171663810 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9480fffc0 msgr2=0x7fc9481023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:12.765 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.762+0000 7fc94d3fa700 1 --2- 172.21.15.176:0/3171663810 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9480fffc0 0x7fc9481023a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc930004940 tx=0x7fc930009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:12.765 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.762+0000 7fc94d3fa700 1 -- 172.21.15.176:0/3171663810 shutdown_connections 2024-03-19T03:12:12.765 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.762+0000 7fc94d3fa700 1 --2- 172.21.15.176:0/3171663810 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9480fffc0 0x7fc9481023a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:12.765 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.762+0000 7fc94d3fa700 1 --2- 172.21.15.176:0/3171663810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc948066e50 0x7fc9480ffa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:12.765 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.762+0000 7fc94d3fa700 1 -- 172.21.15.176:0/3171663810 >> 172.21.15.176:0/3171663810 conn(0x7fc9480f7430 msgr2=0x7fc9480f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:12.765 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.762+0000 7fc94d3fa700 1 -- 172.21.15.176:0/3171663810 shutdown_connections 2024-03-19T03:12:12.765 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.762+0000 7fc94d3fa700 1 -- 172.21.15.176:0/3171663810 wait complete. 2024-03-19T03:12:12.765 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.762+0000 7fc94d3fa700 1 Processor -- start 2024-03-19T03:12:12.765 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.762+0000 7fc94d3fa700 1 -- start start 2024-03-19T03:12:12.766 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.763+0000 7fc94d3fa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc948066e50 0x7fc94810d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:12.766 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.763+0000 7fc94d3fa700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9480fffc0 0x7fc94810dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:12.766 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.763+0000 7fc94d3fa700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94810e4e0 con 0x7fc948066e50 2024-03-19T03:12:12.766 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.763+0000 7fc94d3fa700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94810e650 con 0x7fc9480fffc0 2024-03-19T03:12:12.766 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.763+0000 7fc93e7fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9480fffc0 0x7fc94810dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:12.766 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.763+0000 7fc93e7fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9480fffc0 0x7fc94810dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:45966/0 (socket says 172.21.15.176:45966) 2024-03-19T03:12:12.766 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.763+0000 7fc93e7fc700 1 -- 172.21.15.176:0/1059403992 learned_addr learned my addr 172.21.15.176:0/1059403992 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-03-19T03:12:12.766 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.763+0000 7fc946ffd700 1 --2- 172.21.15.176:0/1059403992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc948066e50 0x7fc94810d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:12.766 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.763+0000 7fc93e7fc700 1 -- 172.21.15.176:0/1059403992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc948066e50 msgr2=0x7fc94810d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:12.766 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.763+0000 7fc93e7fc700 1 --2- 172.21.15.176:0/1059403992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc948066e50 0x7fc94810d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:12.766 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.763+0000 7fc93e7fc700 1 -- 172.21.15.176:0/1059403992 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc930005040 con 0x7fc9480fffc0 2024-03-19T03:12:12.766 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.763+0000 7fc93e7fc700 1 --2- 172.21.15.176:0/1059403992 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9480fffc0 0x7fc94810dee0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc93000b3b0 tx=0x7fc930009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:12.766 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.763+0000 7fc93ffff700 1 -- 172.21.15.176:0/1059403992 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc93000e040 con 0x7fc9480fffc0 2024-03-19T03:12:12.766 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.763+0000 7fc93ffff700 1 -- 172.21.15.176:0/1059403992 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc93000bcc0 con 0x7fc9480fffc0 2024-03-19T03:12:12.767 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.763+0000 7fc94d3fa700 1 -- 172.21.15.176:0/1059403992 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc948113000 con 0x7fc9480fffc0 2024-03-19T03:12:12.767 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.763+0000 7fc93ffff700 1 -- 172.21.15.176:0/1059403992 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc93001c630 con 0x7fc9480fffc0 2024-03-19T03:12:12.767 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.763+0000 7fc94d3fa700 1 -- 172.21.15.176:0/1059403992 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9481135c0 con 0x7fc9480fffc0 2024-03-19T03:12:12.767 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.764+0000 7fc94d3fa700 1 -- 172.21.15.176:0/1059403992 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc94804c710 con 0x7fc9480fffc0 2024-03-19T03:12:12.768 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.765+0000 7fc93ffff700 1 -- 172.21.15.176:0/1059403992 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc9300139f0 con 0x7fc9480fffc0 2024-03-19T03:12:12.768 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.765+0000 7fc93ffff700 1 --2- 172.21.15.176:0/1059403992 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc9340692a0 0x7fc93406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:12.768 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.765+0000 7fc93ffff700 1 -- 172.21.15.176:0/1059403992 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc930087c50 con 0x7fc9480fffc0 2024-03-19T03:12:12.768 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.765+0000 7fc946ffd700 1 --2- 172.21.15.176:0/1059403992 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc9340692a0 0x7fc93406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:12.768 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.766+0000 7fc946ffd700 1 --2- 172.21.15.176:0/1059403992 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc9340692a0 0x7fc93406b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc948066b90 tx=0x7fc93800b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:12.771 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:12.770+0000 7fc93ffff700 1 -- 172.21.15.176:0/1059403992 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fc930057d50 con 0x7fc9480fffc0 2024-03-19T03:12:13.029 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:13.028+0000 7fc94d3fa700 1 -- 172.21.15.176:0/1059403992 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fc9481058c0 con 0x7fc9480fffc0 2024-03-19T03:12:13.035 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:13.034+0000 7fc93ffff700 1 -- 172.21.15.176:0/1059403992 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7fc9300578e0 con 0x7fc9480fffc0 2024-03-19T03:12:13.035 INFO:teuthology.orchestra.run.smithi176.stdout:[client.1] 2024-03-19T03:12:13.035 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCNAvll28jEARAAaS2Qm+4yU0vyfYxXumxjow== 2024-03-19T03:12:13.037 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:13.036+0000 7fc94d3fa700 1 -- 172.21.15.176:0/1059403992 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc9340692a0 msgr2=0x7fc93406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:13.037 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:13.036+0000 7fc94d3fa700 1 --2- 172.21.15.176:0/1059403992 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc9340692a0 0x7fc93406b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc948066b90 tx=0x7fc93800b040 comp rx=0 tx=0).stop 2024-03-19T03:12:13.037 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:13.036+0000 7fc94d3fa700 1 -- 172.21.15.176:0/1059403992 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9480fffc0 msgr2=0x7fc94810dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:13.037 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:13.036+0000 7fc94d3fa700 1 --2- 172.21.15.176:0/1059403992 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9480fffc0 0x7fc94810dee0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc93000b3b0 tx=0x7fc930009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:13.037 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:13.036+0000 7fc94d3fa700 1 -- 172.21.15.176:0/1059403992 shutdown_connections 2024-03-19T03:12:13.038 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:13.036+0000 7fc94d3fa700 1 --2- 172.21.15.176:0/1059403992 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc9340692a0 0x7fc93406b750 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:13.038 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:13.036+0000 7fc94d3fa700 1 --2- 172.21.15.176:0/1059403992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc948066e50 0x7fc94810d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:13.038 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:13.036+0000 7fc94d3fa700 1 --2- 172.21.15.176:0/1059403992 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9480fffc0 0x7fc94810dee0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:13.038 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:13.036+0000 7fc94d3fa700 1 -- 172.21.15.176:0/1059403992 >> 172.21.15.176:0/1059403992 conn(0x7fc9480f7430 msgr2=0x7fc9480f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:13.038 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:13.036+0000 7fc94d3fa700 1 -- 172.21.15.176:0/1059403992 shutdown_connections 2024-03-19T03:12:13.038 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:12:13.036+0000 7fc94d3fa700 1 -- 172.21.15.176:0/1059403992 wait complete. 2024-03-19T03:12:13.624 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-19T03:12:13.625 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-19T03:12:13.625 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-19T03:12:13.666 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-19T03:12:13.666 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-19T03:12:13.667 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-19T03:12:13.933 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:14.291 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:14 smithi059 ceph-mon[29133]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:14.291 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:14 smithi059 ceph-mon[29133]: from='client.? 172.21.15.176:0/1059403992' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T03:12:14.291 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:14 smithi059 ceph-mon[29133]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T03:12:14.291 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:14 smithi059 ceph-mon[29133]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-19T03:12:14.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:14 smithi176 ceph-mon[36450]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:14.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:14 smithi176 ceph-mon[36450]: from='client.? 172.21.15.176:0/1059403992' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T03:12:14.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:14 smithi176 ceph-mon[36450]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T03:12:14.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:14 smithi176 ceph-mon[36450]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-19T03:12:15.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.591+0000 7f671d1fe700 1 -- 172.21.15.59:0/1636256141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67180fe800 msgr2=0x7f6718106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:15.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.591+0000 7f671d1fe700 1 --2- 172.21.15.59:0/1636256141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67180fe800 0x7f6718106ed0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f670c004940 tx=0x7f670c009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:15.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.591+0000 7f671d1fe700 1 -- 172.21.15.59:0/1636256141 shutdown_connections 2024-03-19T03:12:15.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.591+0000 7f671d1fe700 1 --2- 172.21.15.59:0/1636256141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67180fe800 0x7f6718106ed0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:15.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.591+0000 7f671d1fe700 1 --2- 172.21.15.59:0/1636256141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67180fde60 0x7f67180fe230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:15.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.591+0000 7f671d1fe700 1 -- 172.21.15.59:0/1636256141 >> 172.21.15.59:0/1636256141 conn(0x7f67180f78e0 msgr2=0x7f67180f9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:15.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.592+0000 7f671d1fe700 1 -- 172.21.15.59:0/1636256141 shutdown_connections 2024-03-19T03:12:15.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.592+0000 7f671d1fe700 1 -- 172.21.15.59:0/1636256141 wait complete. 2024-03-19T03:12:15.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.592+0000 7f671d1fe700 1 Processor -- start 2024-03-19T03:12:15.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.592+0000 7f671d1fe700 1 -- start start 2024-03-19T03:12:15.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.592+0000 7f671d1fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67180fde60 0x7f67180fbd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:15.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.592+0000 7f671d1fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67180fe800 0x7f67180fc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:15.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.592+0000 7f671d1fe700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67180fdc70 con 0x7f67180fe800 2024-03-19T03:12:15.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.592+0000 7f671d1fe700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67180fc7a0 con 0x7f67180fde60 2024-03-19T03:12:15.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.593+0000 7f671659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67180fe800 0x7f67180fc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:15.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.593+0000 7f671659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67180fe800 0x7f67180fc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58456/0 (socket says 172.21.15.59:58456) 2024-03-19T03:12:15.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.593+0000 7f671659c700 1 -- 172.21.15.59:0/172388800 learned_addr learned my addr 172.21.15.59:0/172388800 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:15.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.593+0000 7f6716d9d700 1 --2- 172.21.15.59:0/172388800 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67180fde60 0x7f67180fbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:15.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.593+0000 7f671659c700 1 -- 172.21.15.59:0/172388800 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67180fde60 msgr2=0x7f67180fbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:15.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.593+0000 7f671659c700 1 --2- 172.21.15.59:0/172388800 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67180fde60 0x7f67180fbd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:15.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.593+0000 7f671659c700 1 -- 172.21.15.59:0/172388800 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f670c005040 con 0x7f67180fe800 2024-03-19T03:12:15.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.593+0000 7f671659c700 1 --2- 172.21.15.59:0/172388800 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67180fe800 0x7f67180fc260 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f670c000c00 tx=0x7f670c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:15.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.593+0000 7f66ff7fe700 1 -- 172.21.15.59:0/172388800 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f670c00e070 con 0x7f67180fe800 2024-03-19T03:12:15.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.593+0000 7f671d1fe700 1 -- 172.21.15.59:0/172388800 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67180fca20 con 0x7f67180fe800 2024-03-19T03:12:15.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.593+0000 7f66ff7fe700 1 -- 172.21.15.59:0/172388800 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f670c013cd0 con 0x7f67180fe800 2024-03-19T03:12:15.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.593+0000 7f66ff7fe700 1 -- 172.21.15.59:0/172388800 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f670c012600 con 0x7f67180fe800 2024-03-19T03:12:15.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.593+0000 7f671d1fe700 1 -- 172.21.15.59:0/172388800 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671811a070 con 0x7f67180fe800 2024-03-19T03:12:15.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.594+0000 7f671d1fe700 1 -- 172.21.15.59:0/172388800 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6718066a90 con 0x7f67180fe800 2024-03-19T03:12:15.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.595+0000 7f66ff7fe700 1 -- 172.21.15.59:0/172388800 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f670c01c960 con 0x7f67180fe800 2024-03-19T03:12:15.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.595+0000 7f66ff7fe700 1 --2- 172.21.15.59:0/172388800 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f67000692a0 0x7f670006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:15.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.595+0000 7f6716d9d700 1 --2- 172.21.15.59:0/172388800 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f67000692a0 0x7f670006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:15.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.595+0000 7f66ff7fe700 1 -- 172.21.15.59:0/172388800 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f670c088750 con 0x7f67180fe800 2024-03-19T03:12:15.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.595+0000 7f6716d9d700 1 --2- 172.21.15.59:0/172388800 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f67000692a0 0x7f670006b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6708004600 tx=0x7f670800b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:15.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.600+0000 7f66ff7fe700 1 -- 172.21.15.59:0/172388800 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f670c054dc0 con 0x7f67180fe800 2024-03-19T03:12:15.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.847+0000 7f671d1fe700 1 -- 172.21.15.59:0/172388800 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f671804c710 con 0x7f67180fe800 2024-03-19T03:12:15.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.850+0000 7f66ff7fe700 1 -- 172.21.15.59:0/172388800 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167168 (secure 0 0 0) 0x7f670c004490 con 0x7f67180fe800 2024-03-19T03:12:15.855 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:12:15.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.853+0000 7f671d1fe700 1 -- 172.21.15.59:0/172388800 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f67000692a0 msgr2=0x7f670006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:15.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.853+0000 7f671d1fe700 1 --2- 172.21.15.59:0/172388800 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f67000692a0 0x7f670006b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6708004600 tx=0x7f670800b040 comp rx=0 tx=0).stop 2024-03-19T03:12:15.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.853+0000 7f671d1fe700 1 -- 172.21.15.59:0/172388800 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67180fe800 msgr2=0x7f67180fc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:15.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.853+0000 7f671d1fe700 1 --2- 172.21.15.59:0/172388800 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67180fe800 0x7f67180fc260 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f670c000c00 tx=0x7f670c009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:15.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.853+0000 7f671d1fe700 1 -- 172.21.15.59:0/172388800 shutdown_connections 2024-03-19T03:12:15.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.853+0000 7f671d1fe700 1 --2- 172.21.15.59:0/172388800 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f67000692a0 0x7f670006b750 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:15.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.853+0000 7f671d1fe700 1 --2- 172.21.15.59:0/172388800 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67180fde60 0x7f67180fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:15.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.853+0000 7f671d1fe700 1 --2- 172.21.15.59:0/172388800 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67180fe800 0x7f67180fc260 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:15.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.853+0000 7f671d1fe700 1 -- 172.21.15.59:0/172388800 >> 172.21.15.59:0/172388800 conn(0x7f67180f78e0 msgr2=0x7f6718103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:15.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.853+0000 7f671d1fe700 1 -- 172.21.15.59:0/172388800 shutdown_connections 2024-03-19T03:12:15.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:15.853+0000 7f671d1fe700 1 -- 172.21.15.59:0/172388800 wait complete. 2024-03-19T03:12:16.236 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:16 smithi059 ceph-mon[29133]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:16.236 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:16 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/172388800' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-19T03:12:16.236 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:16 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:12:16.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:16 smithi176 ceph-mon[36450]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:16.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:16 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/172388800' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-19T03:12:16.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:16 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:12:16.630 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":19,"active_gid":14214,"active_name":"smithi059.azpecc","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2563034496},{"type":"v1","addr":"172.21.15.59:6801","nonce":2563034496}]},"active_addr":"172.21.15.59:6801/2563034496","active_change":"2024-03-19T03:08:30.903242+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14234,"name":"smithi176.ycmqbw","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.59:8443/","prometheus":"http://172.21.15.59:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.59:0","nonce":2512718707}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.59:0","nonce":2111540519}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.59:0","nonce":3717556970}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.59:0","nonce":193121325}]}]} 2024-03-19T03:12:16.635 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-19T03:12:16.635 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-19T03:12:16.635 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-19T03:12:16.895 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:18.130 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:18 smithi059 ceph-mon[29133]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:18.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:18 smithi176 ceph-mon[36450]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:18.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.403+0000 7fd4d4409700 1 -- 172.21.15.59:0/621799436 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4cc102e10 msgr2=0x7fd4cc1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:18.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.403+0000 7fd4d4409700 1 --2- 172.21.15.59:0/621799436 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4cc102e10 0x7fd4cc1031e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc009ce0 tx=0x7fd4bc004d10 comp rx=0 tx=0).stop 2024-03-19T03:12:18.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.403+0000 7fd4d4409700 1 -- 172.21.15.59:0/621799436 shutdown_connections 2024-03-19T03:12:18.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.403+0000 7fd4d4409700 1 --2- 172.21.15.59:0/621799436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4cc1037b0 0x7fd4cc0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:18.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.403+0000 7fd4d4409700 1 --2- 172.21.15.59:0/621799436 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4cc102e10 0x7fd4cc1031e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:18.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.403+0000 7fd4d4409700 1 -- 172.21.15.59:0/621799436 >> 172.21.15.59:0/621799436 conn(0x7fd4cc0f8980 msgr2=0x7fd4cc0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:18.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.403+0000 7fd4d4409700 1 -- 172.21.15.59:0/621799436 shutdown_connections 2024-03-19T03:12:18.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.403+0000 7fd4d4409700 1 -- 172.21.15.59:0/621799436 wait complete. 2024-03-19T03:12:18.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.403+0000 7fd4d4409700 1 Processor -- start 2024-03-19T03:12:18.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.404+0000 7fd4d4409700 1 -- start start 2024-03-19T03:12:18.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.404+0000 7fd4d4409700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4cc102e10 0x7fd4cc113d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:18.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.404+0000 7fd4d4409700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4cc1037b0 0x7fd4cc114260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:18.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.404+0000 7fd4d4409700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc114940 con 0x7fd4cc1037b0 2024-03-19T03:12:18.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.404+0000 7fd4d4409700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc10dc80 con 0x7fd4cc102e10 2024-03-19T03:12:18.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.404+0000 7fd4d19a4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4cc1037b0 0x7fd4cc114260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:18.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.404+0000 7fd4d19a4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4cc1037b0 0x7fd4cc114260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58468/0 (socket says 172.21.15.59:58468) 2024-03-19T03:12:18.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.404+0000 7fd4d19a4700 1 -- 172.21.15.59:0/862092991 learned_addr learned my addr 172.21.15.59:0/862092991 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:18.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.404+0000 7fd4d21a5700 1 --2- 172.21.15.59:0/862092991 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4cc102e10 0x7fd4cc113d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:18.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.404+0000 7fd4d19a4700 1 -- 172.21.15.59:0/862092991 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4cc102e10 msgr2=0x7fd4cc113d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:18.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.404+0000 7fd4d19a4700 1 --2- 172.21.15.59:0/862092991 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4cc102e10 0x7fd4cc113d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:18.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.404+0000 7fd4d19a4700 1 -- 172.21.15.59:0/862092991 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4bc009940 con 0x7fd4cc1037b0 2024-03-19T03:12:18.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.404+0000 7fd4d19a4700 1 --2- 172.21.15.59:0/862092991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4cc1037b0 0x7fd4cc114260 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd4c000b790 tx=0x7fd4c0009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:18.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.405+0000 7fd4caffd700 1 -- 172.21.15.59:0/862092991 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4c000f6f0 con 0x7fd4cc1037b0 2024-03-19T03:12:18.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.405+0000 7fd4caffd700 1 -- 172.21.15.59:0/862092991 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd4c000fd20 con 0x7fd4cc1037b0 2024-03-19T03:12:18.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.405+0000 7fd4d4409700 1 -- 172.21.15.59:0/862092991 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4cc10df00 con 0x7fd4cc1037b0 2024-03-19T03:12:18.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.405+0000 7fd4caffd700 1 -- 172.21.15.59:0/862092991 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4c00103a0 con 0x7fd4cc1037b0 2024-03-19T03:12:18.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.405+0000 7fd4d4409700 1 -- 172.21.15.59:0/862092991 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4cc10e470 con 0x7fd4cc1037b0 2024-03-19T03:12:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.406+0000 7fd4d4409700 1 -- 172.21.15.59:0/862092991 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4cc0fe560 con 0x7fd4cc1037b0 2024-03-19T03:12:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.406+0000 7fd4caffd700 1 -- 172.21.15.59:0/862092991 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd4c0011400 con 0x7fd4cc1037b0 2024-03-19T03:12:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.406+0000 7fd4caffd700 1 --2- 172.21.15.59:0/862092991 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd4b40692a0 0x7fd4b406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.407+0000 7fd4d21a5700 1 --2- 172.21.15.59:0/862092991 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd4b40692a0 0x7fd4b406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.407+0000 7fd4caffd700 1 -- 172.21.15.59:0/862092991 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd4c0086c20 con 0x7fd4cc1037b0 2024-03-19T03:12:18.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.407+0000 7fd4d21a5700 1 --2- 172.21.15.59:0/862092991 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd4b40692a0 0x7fd4b406b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc00b740 tx=0x7fd4bc004d10 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:18.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.411+0000 7fd4caffd700 1 -- 172.21.15.59:0/862092991 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fd4c0056d20 con 0x7fd4cc1037b0 2024-03-19T03:12:18.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.602+0000 7fd4d4409700 1 -- 172.21.15.59:0/862092991 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4cc05f5f0 con 0x7fd4cc1037b0 2024-03-19T03:12:18.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.603+0000 7fd4caffd700 1 -- 172.21.15.59:0/862092991 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11081 (secure 0 0 0) 0x7fd4c00568b0 con 0x7fd4cc1037b0 2024-03-19T03:12:18.604 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:12:18.604 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":40,"fsid":"8c8ae530-e59d-11ee-95c9-87774f69a715","created":"2024-03-19T03:06:15.064131+0000","modified":"2024-03-19T03:11:59.268254+0000","last_up_change":"2024-03-19T03:11:58.260172+0000","last_in_change":"2024-03-19T03:11:40.023566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:10:45.238939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"f6a12d03-3dd9-40d4-b54e-4b2041c18e48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1770897334},{"type":"v1","addr":"172.21.15.59:6803","nonce":1770897334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1770897334},{"type":"v1","addr":"172.21.15.59:6805","nonce":1770897334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1770897334},{"type":"v1","addr":"172.21.15.59:6809","nonce":1770897334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1770897334},{"type":"v1","addr":"172.21.15.59:6807","nonce":1770897334}]},"public_addr":"172.21.15.59:6803/1770897334","cluster_addr":"172.21.15.59:6805/1770897334","heartbeat_back_addr":"172.21.15.59:6809/1770897334","heartbeat_front_addr":"172.21.15.59:6807/1770897334","state":["exists","up"]},{"osd":1,"uuid":"c1069548-eb23-476b-a661-b71f0a73987e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3595398218},{"type":"v1","addr":"172.21.15.59:6811","nonce":3595398218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3595398218},{"type":"v1","addr":"172.21.15.59:6813","nonce":3595398218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":3595398218},{"type":"v1","addr":"172.21.15.59:6817","nonce":3595398218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3595398218},{"type":"v1","addr":"172.21.15.59:6815","nonce":3595398218}]},"public_addr":"172.21.15.59:6811/3595398218","cluster_addr":"172.21.15.59:6813/3595398218","heartbeat_back_addr":"172.21.15.59:6817/3595398218","heartbeat_front_addr":"172.21.15.59:6815/3595398218","state":["exists","up"]},{"osd":2,"uuid":"4958f1e4-2dc9-458f-82ff-e51d98762cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":639205676},{"type":"v1","addr":"172.21.15.59:6819","nonce":639205676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":639205676},{"type":"v1","addr":"172.21.15.59:6821","nonce":639205676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":639205676},{"type":"v1","addr":"172.21.15.59:6825","nonce":639205676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":639205676},{"type":"v1","addr":"172.21.15.59:6823","nonce":639205676}]},"public_addr":"172.21.15.59:6819/639205676","cluster_addr":"172.21.15.59:6821/639205676","heartbeat_back_addr":"172.21.15.59:6825/639205676","heartbeat_front_addr":"172.21.15.59:6823/639205676","state":["exists","up"]},{"osd":3,"uuid":"c3cf9095-7ce2-4d0f-bc65-2cda09504d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":3291292674},{"type":"v1","addr":"172.21.15.176:6801","nonce":3291292674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":3291292674},{"type":"v1","addr":"172.21.15.176:6803","nonce":3291292674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":3291292674},{"type":"v1","addr":"172.21.15.176:6807","nonce":3291292674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":3291292674},{"type":"v1","addr":"172.21.15.176:6805","nonce":3291292674}]},"public_addr":"172.21.15.176:6801/3291292674","cluster_addr":"172.21.15.176:6803/3291292674","heartbeat_back_addr":"172.21.15.176:6807/3291292674","heartbeat_front_addr":"172.21.15.176:6805/3291292674","state":["exists","up"]},{"osd":4,"uuid":"b102d232-9f65-4d96-ad81-b3d1a0b8faf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":1006246702},{"type":"v1","addr":"172.21.15.176:6809","nonce":1006246702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":1006246702},{"type":"v1","addr":"172.21.15.176:6811","nonce":1006246702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":1006246702},{"type":"v1","addr":"172.21.15.176:6815","nonce":1006246702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":1006246702},{"type":"v1","addr":"172.21.15.176:6813","nonce":1006246702}]},"public_addr":"172.21.15.176:6809/1006246702","cluster_addr":"172.21.15.176:6811/1006246702","heartbeat_back_addr":"172.21.15.176:6815/1006246702","heartbeat_front_addr":"172.21.15.176:6813/1006246702","state":["exists","up"]},{"osd":5,"uuid":"f9c805ed-8aa0-4f39-a78f-c6a4b018eb7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":3279585260},{"type":"v1","addr":"172.21.15.176:6817","nonce":3279585260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":3279585260},{"type":"v1","addr":"172.21.15.176:6819","nonce":3279585260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":3279585260},{"type":"v1","addr":"172.21.15.176:6823","nonce":3279585260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":3279585260},{"type":"v1","addr":"172.21.15.176:6821","nonce":3279585260}]},"public_addr":"172.21.15.176:6817/3279585260","cluster_addr":"172.21.15.176:6819/3279585260","heartbeat_back_addr":"172.21.15.176:6823/3279585260","heartbeat_front_addr":"172.21.15.176:6821/3279585260","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:09:44.576635+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:10:12.913840+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:10:42.022990+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:11:05.039788+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:11:31.198904+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:11:57.135656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:0/2117971082":"2024-03-20T03:08:30.903133+0000","172.21.15.59:0/988882608":"2024-03-20T03:07:20.738216+0000","172.21.15.59:0/2884557489":"2024-03-20T03:08:30.903133+0000","172.21.15.59:6801/1573642037":"2024-03-20T03:07:20.738216+0000","172.21.15.59:0/2956286237":"2024-03-20T03:07:20.738216+0000","172.21.15.59:6801/1857243897":"2024-03-20T03:08:30.903133+0000","172.21.15.59:0/661762888":"2024-03-20T03:06:49.795873+0000","172.21.15.59:0/2659339884":"2024-03-20T03:06:49.795873+0000","172.21.15.59:0/156375080":"2024-03-20T03:06:49.795873+0000","172.21.15.59:6800/3609017287":"2024-03-20T03:06:49.795873+0000","172.21.15.59:0/133172490":"2024-03-20T03:08:30.903133+0000","172.21.15.59:6800/1573642037":"2024-03-20T03:07:20.738216+0000","172.21.15.59:6801/3609017287":"2024-03-20T03:06:49.795873+0000","172.21.15.59:6800/1857243897":"2024-03-20T03:08:30.903133+0000","172.21.15.59:0/2483030805":"2024-03-20T03:07:20.738216+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:12:18.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.605+0000 7fd4d4409700 1 -- 172.21.15.59:0/862092991 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd4b40692a0 msgr2=0x7fd4b406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:18.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.605+0000 7fd4d4409700 1 --2- 172.21.15.59:0/862092991 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd4b40692a0 0x7fd4b406b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc00b740 tx=0x7fd4bc004d10 comp rx=0 tx=0).stop 2024-03-19T03:12:18.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.605+0000 7fd4d4409700 1 -- 172.21.15.59:0/862092991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4cc1037b0 msgr2=0x7fd4cc114260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:18.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.605+0000 7fd4d4409700 1 --2- 172.21.15.59:0/862092991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4cc1037b0 0x7fd4cc114260 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd4c000b790 tx=0x7fd4c0009220 comp rx=0 tx=0).stop 2024-03-19T03:12:18.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.605+0000 7fd4d4409700 1 -- 172.21.15.59:0/862092991 shutdown_connections 2024-03-19T03:12:18.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.605+0000 7fd4d4409700 1 --2- 172.21.15.59:0/862092991 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd4b40692a0 0x7fd4b406b750 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:18.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.605+0000 7fd4d4409700 1 --2- 172.21.15.59:0/862092991 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4cc102e10 0x7fd4cc113d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:18.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.605+0000 7fd4d4409700 1 --2- 172.21.15.59:0/862092991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4cc1037b0 0x7fd4cc114260 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:18.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.605+0000 7fd4d4409700 1 -- 172.21.15.59:0/862092991 >> 172.21.15.59:0/862092991 conn(0x7fd4cc0f8980 msgr2=0x7fd4cc0f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:18.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.605+0000 7fd4d4409700 1 -- 172.21.15.59:0/862092991 shutdown_connections 2024-03-19T03:12:18.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:18.605+0000 7fd4d4409700 1 -- 172.21.15.59:0/862092991 wait complete. 2024-03-19T03:12:19.129 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:19 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/862092991' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:12:19.198 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-19T03:12:19.198 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-19T03:12:19.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:19 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/862092991' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:12:19.457 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:20.214 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:20 smithi059 ceph-mon[29133]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:20.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:20 smithi176 ceph-mon[36450]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:20.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.942+0000 7f32fd8eb700 1 -- 172.21.15.59:0/236054118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f80fddd0 msgr2=0x7f32f80fe1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:20.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.942+0000 7f32fd8eb700 1 --2- 172.21.15.59:0/236054118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f80fddd0 0x7f32f80fe1a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f32e8009ce0 tx=0x7f32e8004d10 comp rx=0 tx=0).stop 2024-03-19T03:12:20.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.942+0000 7f32fd8eb700 1 -- 172.21.15.59:0/236054118 shutdown_connections 2024-03-19T03:12:20.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.942+0000 7f32fd8eb700 1 --2- 172.21.15.59:0/236054118 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32f80fe770 0x7f32f8106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:20.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.942+0000 7f32fd8eb700 1 --2- 172.21.15.59:0/236054118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f80fddd0 0x7f32f80fe1a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:20.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.942+0000 7f32fd8eb700 1 -- 172.21.15.59:0/236054118 >> 172.21.15.59:0/236054118 conn(0x7f32f80f7890 msgr2=0x7f32f80f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:20.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.942+0000 7f32fd8eb700 1 -- 172.21.15.59:0/236054118 shutdown_connections 2024-03-19T03:12:20.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.942+0000 7f32fd8eb700 1 -- 172.21.15.59:0/236054118 wait complete. 2024-03-19T03:12:20.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.943+0000 7f32fd8eb700 1 Processor -- start 2024-03-19T03:12:20.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.943+0000 7f32fd8eb700 1 -- start start 2024-03-19T03:12:20.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.943+0000 7f32fd8eb700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32f80fddd0 0x7f32f8113900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:20.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.943+0000 7f32fd8eb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f80fe770 0x7f32f8113e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:20.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.943+0000 7f32fd8eb700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f8114520 con 0x7f32f80fe770 2024-03-19T03:12:20.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.943+0000 7f32fd8eb700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f810da90 con 0x7f32f80fddd0 2024-03-19T03:12:20.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.943+0000 7f32f67fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f80fe770 0x7f32f8113e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:20.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.943+0000 7f32f67fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f80fe770 0x7f32f8113e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58486/0 (socket says 172.21.15.59:58486) 2024-03-19T03:12:20.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.943+0000 7f32f67fc700 1 -- 172.21.15.59:0/758539458 learned_addr learned my addr 172.21.15.59:0/758539458 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:20.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.943+0000 7f32f6ffd700 1 --2- 172.21.15.59:0/758539458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32f80fddd0 0x7f32f8113900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:20.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.943+0000 7f32f67fc700 1 -- 172.21.15.59:0/758539458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32f80fddd0 msgr2=0x7f32f8113900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:20.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.944+0000 7f32f67fc700 1 --2- 172.21.15.59:0/758539458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32f80fddd0 0x7f32f8113900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:20.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.944+0000 7f32f67fc700 1 -- 172.21.15.59:0/758539458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e8009940 con 0x7f32f80fe770 2024-03-19T03:12:20.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.944+0000 7f32f67fc700 1 --2- 172.21.15.59:0/758539458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f80fe770 0x7f32f8113e40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f32ec00b790 tx=0x7f32ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:20.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.944+0000 7f32df7fe700 1 -- 172.21.15.59:0/758539458 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32ec00f6f0 con 0x7f32f80fe770 2024-03-19T03:12:20.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.944+0000 7f32fd8eb700 1 -- 172.21.15.59:0/758539458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f810dd70 con 0x7f32f80fe770 2024-03-19T03:12:20.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.944+0000 7f32fd8eb700 1 -- 172.21.15.59:0/758539458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f810e2b0 con 0x7f32f80fe770 2024-03-19T03:12:20.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.944+0000 7f32df7fe700 1 -- 172.21.15.59:0/758539458 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f32ec00fd20 con 0x7f32f80fe770 2024-03-19T03:12:20.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.944+0000 7f32df7fe700 1 -- 172.21.15.59:0/758539458 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32ec0103a0 con 0x7f32f80fe770 2024-03-19T03:12:20.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.945+0000 7f32fd8eb700 1 -- 172.21.15.59:0/758539458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f8066b50 con 0x7f32f80fe770 2024-03-19T03:12:20.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.945+0000 7f32df7fe700 1 -- 172.21.15.59:0/758539458 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f32ec011400 con 0x7f32f80fe770 2024-03-19T03:12:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.946+0000 7f32df7fe700 1 --2- 172.21.15.59:0/758539458 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f32e00692a0 0x7f32e006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.946+0000 7f32f6ffd700 1 --2- 172.21.15.59:0/758539458 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f32e00692a0 0x7f32e006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.946+0000 7f32df7fe700 1 -- 172.21.15.59:0/758539458 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f32ec086b50 con 0x7f32f80fe770 2024-03-19T03:12:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.946+0000 7f32f6ffd700 1 --2- 172.21.15.59:0/758539458 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f32e00692a0 0x7f32e006b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f32e800b740 tx=0x7f32e8004d10 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:20.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:20.950+0000 7f32df7fe700 1 -- 172.21.15.59:0/758539458 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f32ec056cf0 con 0x7f32f80fe770 2024-03-19T03:12:21.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:21.142+0000 7f32fd8eb700 1 -- 172.21.15.59:0/758539458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32f804c710 con 0x7f32f80fe770 2024-03-19T03:12:21.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:21.143+0000 7f32df7fe700 1 -- 172.21.15.59:0/758539458 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11081 (secure 0 0 0) 0x7f32ec056880 con 0x7f32f80fe770 2024-03-19T03:12:21.144 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:12:21.144 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":40,"fsid":"8c8ae530-e59d-11ee-95c9-87774f69a715","created":"2024-03-19T03:06:15.064131+0000","modified":"2024-03-19T03:11:59.268254+0000","last_up_change":"2024-03-19T03:11:58.260172+0000","last_in_change":"2024-03-19T03:11:40.023566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:10:45.238939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"f6a12d03-3dd9-40d4-b54e-4b2041c18e48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1770897334},{"type":"v1","addr":"172.21.15.59:6803","nonce":1770897334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1770897334},{"type":"v1","addr":"172.21.15.59:6805","nonce":1770897334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1770897334},{"type":"v1","addr":"172.21.15.59:6809","nonce":1770897334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1770897334},{"type":"v1","addr":"172.21.15.59:6807","nonce":1770897334}]},"public_addr":"172.21.15.59:6803/1770897334","cluster_addr":"172.21.15.59:6805/1770897334","heartbeat_back_addr":"172.21.15.59:6809/1770897334","heartbeat_front_addr":"172.21.15.59:6807/1770897334","state":["exists","up"]},{"osd":1,"uuid":"c1069548-eb23-476b-a661-b71f0a73987e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3595398218},{"type":"v1","addr":"172.21.15.59:6811","nonce":3595398218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3595398218},{"type":"v1","addr":"172.21.15.59:6813","nonce":3595398218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":3595398218},{"type":"v1","addr":"172.21.15.59:6817","nonce":3595398218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3595398218},{"type":"v1","addr":"172.21.15.59:6815","nonce":3595398218}]},"public_addr":"172.21.15.59:6811/3595398218","cluster_addr":"172.21.15.59:6813/3595398218","heartbeat_back_addr":"172.21.15.59:6817/3595398218","heartbeat_front_addr":"172.21.15.59:6815/3595398218","state":["exists","up"]},{"osd":2,"uuid":"4958f1e4-2dc9-458f-82ff-e51d98762cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":639205676},{"type":"v1","addr":"172.21.15.59:6819","nonce":639205676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":639205676},{"type":"v1","addr":"172.21.15.59:6821","nonce":639205676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":639205676},{"type":"v1","addr":"172.21.15.59:6825","nonce":639205676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":639205676},{"type":"v1","addr":"172.21.15.59:6823","nonce":639205676}]},"public_addr":"172.21.15.59:6819/639205676","cluster_addr":"172.21.15.59:6821/639205676","heartbeat_back_addr":"172.21.15.59:6825/639205676","heartbeat_front_addr":"172.21.15.59:6823/639205676","state":["exists","up"]},{"osd":3,"uuid":"c3cf9095-7ce2-4d0f-bc65-2cda09504d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":3291292674},{"type":"v1","addr":"172.21.15.176:6801","nonce":3291292674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":3291292674},{"type":"v1","addr":"172.21.15.176:6803","nonce":3291292674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":3291292674},{"type":"v1","addr":"172.21.15.176:6807","nonce":3291292674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":3291292674},{"type":"v1","addr":"172.21.15.176:6805","nonce":3291292674}]},"public_addr":"172.21.15.176:6801/3291292674","cluster_addr":"172.21.15.176:6803/3291292674","heartbeat_back_addr":"172.21.15.176:6807/3291292674","heartbeat_front_addr":"172.21.15.176:6805/3291292674","state":["exists","up"]},{"osd":4,"uuid":"b102d232-9f65-4d96-ad81-b3d1a0b8faf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":1006246702},{"type":"v1","addr":"172.21.15.176:6809","nonce":1006246702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":1006246702},{"type":"v1","addr":"172.21.15.176:6811","nonce":1006246702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":1006246702},{"type":"v1","addr":"172.21.15.176:6815","nonce":1006246702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":1006246702},{"type":"v1","addr":"172.21.15.176:6813","nonce":1006246702}]},"public_addr":"172.21.15.176:6809/1006246702","cluster_addr":"172.21.15.176:6811/1006246702","heartbeat_back_addr":"172.21.15.176:6815/1006246702","heartbeat_front_addr":"172.21.15.176:6813/1006246702","state":["exists","up"]},{"osd":5,"uuid":"f9c805ed-8aa0-4f39-a78f-c6a4b018eb7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":3279585260},{"type":"v1","addr":"172.21.15.176:6817","nonce":3279585260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":3279585260},{"type":"v1","addr":"172.21.15.176:6819","nonce":3279585260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":3279585260},{"type":"v1","addr":"172.21.15.176:6823","nonce":3279585260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":3279585260},{"type":"v1","addr":"172.21.15.176:6821","nonce":3279585260}]},"public_addr":"172.21.15.176:6817/3279585260","cluster_addr":"172.21.15.176:6819/3279585260","heartbeat_back_addr":"172.21.15.176:6823/3279585260","heartbeat_front_addr":"172.21.15.176:6821/3279585260","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:09:44.576635+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:10:12.913840+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:10:42.022990+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:11:05.039788+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:11:31.198904+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:11:57.135656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:0/2117971082":"2024-03-20T03:08:30.903133+0000","172.21.15.59:0/988882608":"2024-03-20T03:07:20.738216+0000","172.21.15.59:0/2884557489":"2024-03-20T03:08:30.903133+0000","172.21.15.59:6801/1573642037":"2024-03-20T03:07:20.738216+0000","172.21.15.59:0/2956286237":"2024-03-20T03:07:20.738216+0000","172.21.15.59:6801/1857243897":"2024-03-20T03:08:30.903133+0000","172.21.15.59:0/661762888":"2024-03-20T03:06:49.795873+0000","172.21.15.59:0/2659339884":"2024-03-20T03:06:49.795873+0000","172.21.15.59:0/156375080":"2024-03-20T03:06:49.795873+0000","172.21.15.59:6800/3609017287":"2024-03-20T03:06:49.795873+0000","172.21.15.59:0/133172490":"2024-03-20T03:08:30.903133+0000","172.21.15.59:6800/1573642037":"2024-03-20T03:07:20.738216+0000","172.21.15.59:6801/3609017287":"2024-03-20T03:06:49.795873+0000","172.21.15.59:6800/1857243897":"2024-03-20T03:08:30.903133+0000","172.21.15.59:0/2483030805":"2024-03-20T03:07:20.738216+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:12:21.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:21.145+0000 7f32fd8eb700 1 -- 172.21.15.59:0/758539458 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f32e00692a0 msgr2=0x7f32e006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:21.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:21.145+0000 7f32fd8eb700 1 --2- 172.21.15.59:0/758539458 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f32e00692a0 0x7f32e006b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f32e800b740 tx=0x7f32e8004d10 comp rx=0 tx=0).stop 2024-03-19T03:12:21.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:21.145+0000 7f32fd8eb700 1 -- 172.21.15.59:0/758539458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f80fe770 msgr2=0x7f32f8113e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:21.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:21.145+0000 7f32fd8eb700 1 --2- 172.21.15.59:0/758539458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f80fe770 0x7f32f8113e40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f32ec00b790 tx=0x7f32ec009250 comp rx=0 tx=0).stop 2024-03-19T03:12:21.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:21.145+0000 7f32fd8eb700 1 -- 172.21.15.59:0/758539458 shutdown_connections 2024-03-19T03:12:21.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:21.145+0000 7f32fd8eb700 1 --2- 172.21.15.59:0/758539458 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f32e00692a0 0x7f32e006b750 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:21.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:21.145+0000 7f32fd8eb700 1 --2- 172.21.15.59:0/758539458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32f80fddd0 0x7f32f8113900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:21.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:21.145+0000 7f32fd8eb700 1 --2- 172.21.15.59:0/758539458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f80fe770 0x7f32f8113e40 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:21.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:21.145+0000 7f32fd8eb700 1 -- 172.21.15.59:0/758539458 >> 172.21.15.59:0/758539458 conn(0x7f32f80f7890 msgr2=0x7f32f8103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:21.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:21.145+0000 7f32fd8eb700 1 -- 172.21.15.59:0/758539458 shutdown_connections 2024-03-19T03:12:21.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:21.145+0000 7f32fd8eb700 1 -- 172.21.15.59:0/758539458 wait complete. 2024-03-19T03:12:21.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:21 smithi059 ceph-mon[29133]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:21.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:21 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/758539458' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:12:21.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:21 smithi176 ceph-mon[36450]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:21.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:21 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/758539458' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:12:21.821 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-19T03:12:21.821 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-19T03:12:21.821 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-19T03:12:21.822 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-19T03:12:21.822 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-19T03:12:21.823 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-19T03:12:22.123 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:22.132 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:24.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:23 smithi059 ceph-mon[29133]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:24.170 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:24.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:24 smithi176 ceph-mon[36450]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:24.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.899+0000 7f2c88d95700 1 -- 172.21.15.59:0/3612113123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c840fe830 msgr2=0x7f2c84106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:24.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.899+0000 7f2c88d95700 1 --2- 172.21.15.59:0/3612113123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c840fe830 0x7f2c84106f00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c004940 tx=0x7f2c6c009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:24.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.899+0000 7f2c88d95700 1 -- 172.21.15.59:0/3612113123 shutdown_connections 2024-03-19T03:12:24.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.899+0000 7f2c88d95700 1 --2- 172.21.15.59:0/3612113123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c840fe830 0x7f2c84106f00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:24.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.899+0000 7f2c88d95700 1 --2- 172.21.15.59:0/3612113123 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c840fde90 0x7f2c840fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:24.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.899+0000 7f2c88d95700 1 -- 172.21.15.59:0/3612113123 >> 172.21.15.59:0/3612113123 conn(0x7f2c84072bc0 msgr2=0x7f2c84072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:24.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.899+0000 7f2c88d95700 1 -- 172.21.15.59:0/3612113123 shutdown_connections 2024-03-19T03:12:24.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.899+0000 7f2c88d95700 1 -- 172.21.15.59:0/3612113123 wait complete. 2024-03-19T03:12:24.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.900+0000 7f2c88d95700 1 Processor -- start 2024-03-19T03:12:24.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.900+0000 7f2c88d95700 1 -- start start 2024-03-19T03:12:24.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.900+0000 7f2c88d95700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c840fde90 0x7f2c84113c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:24.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.900+0000 7f2c88d95700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c840fe830 0x7f2c84114180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:24.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.900+0000 7f2c88d95700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c841147d0 con 0x7f2c840fe830 2024-03-19T03:12:24.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.900+0000 7f2c88d95700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8410dba0 con 0x7f2c840fde90 2024-03-19T03:12:24.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.900+0000 7f2c81d9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c840fe830 0x7f2c84114180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:24.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.900+0000 7f2c81d9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c840fe830 0x7f2c84114180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36582/0 (socket says 172.21.15.59:36582) 2024-03-19T03:12:24.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.900+0000 7f2c81d9b700 1 -- 172.21.15.59:0/3335620776 learned_addr learned my addr 172.21.15.59:0/3335620776 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:24.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.900+0000 7f2c8259c700 1 --2- 172.21.15.59:0/3335620776 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c840fde90 0x7f2c84113c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:24.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.900+0000 7f2c81d9b700 1 -- 172.21.15.59:0/3335620776 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c840fde90 msgr2=0x7f2c84113c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:24.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.900+0000 7f2c81d9b700 1 --2- 172.21.15.59:0/3335620776 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c840fde90 0x7f2c84113c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:24.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.900+0000 7f2c81d9b700 1 -- 172.21.15.59:0/3335620776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6c005040 con 0x7f2c840fe830 2024-03-19T03:12:24.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.900+0000 7f2c8259c700 1 --2- 172.21.15.59:0/3335620776 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c840fde90 0x7f2c84113c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:12:24.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.901+0000 7f2c81d9b700 1 --2- 172.21.15.59:0/3335620776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c840fe830 0x7f2c84114180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c0093b0 tx=0x7f2c6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:24.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.901+0000 7f2c7affd700 1 -- 172.21.15.59:0/3335620776 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c6c00e040 con 0x7f2c840fe830 2024-03-19T03:12:24.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.901+0000 7f2c7affd700 1 -- 172.21.15.59:0/3335620776 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c6c004290 con 0x7f2c840fe830 2024-03-19T03:12:24.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.901+0000 7f2c88d95700 1 -- 172.21.15.59:0/3335620776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c8410de20 con 0x7f2c840fe830 2024-03-19T03:12:24.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.901+0000 7f2c7affd700 1 -- 172.21.15.59:0/3335620776 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c6c01c3f0 con 0x7f2c840fe830 2024-03-19T03:12:24.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.901+0000 7f2c88d95700 1 -- 172.21.15.59:0/3335620776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c8410e360 con 0x7f2c840fe830 2024-03-19T03:12:24.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.902+0000 7f2c88d95700 1 -- 172.21.15.59:0/3335620776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2c8410e880 con 0x7f2c840fe830 2024-03-19T03:12:24.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.902+0000 7f2c7affd700 1 -- 172.21.15.59:0/3335620776 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f2c6c013890 con 0x7f2c840fe830 2024-03-19T03:12:24.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.903+0000 7f2c7affd700 1 --2- 172.21.15.59:0/3335620776 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2c700692b0 0x7f2c7006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:24.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.903+0000 7f2c8259c700 1 --2- 172.21.15.59:0/3335620776 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2c700692b0 0x7f2c7006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:24.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.903+0000 7f2c7affd700 1 -- 172.21.15.59:0/3335620776 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2c6c0880f0 con 0x7f2c840fe830 2024-03-19T03:12:24.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.903+0000 7f2c7affd700 1 --2- 172.21.15.59:0/3335620776 >> [v2:172.21.15.176:6800/3291292674,v1:172.21.15.176:6801/3291292674] conn(0x7f2c7006ef80 0x7f2c70071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:24.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.903+0000 7f2c7affd700 1 -- 172.21.15.59:0/3335620776 --> [v2:172.21.15.176:6800/3291292674,v1:172.21.15.176:6801/3291292674] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c70071a40 con 0x7f2c7006ef80 2024-03-19T03:12:24.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.903+0000 7f2c7affd700 1 -- 172.21.15.59:0/3335620776 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c6c08a6e0 con 0x7f2c840fe830 2024-03-19T03:12:24.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.903+0000 7f2c8259c700 1 --2- 172.21.15.59:0/3335620776 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2c700692b0 0x7f2c7006b760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2c74004550 tx=0x7f2c74009040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:24.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.903+0000 7f2c82d9d700 1 --2- 172.21.15.59:0/3335620776 >> [v2:172.21.15.176:6800/3291292674,v1:172.21.15.176:6801/3291292674] conn(0x7f2c7006ef80 0x7f2c70071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.904+0000 7f2c82d9d700 1 --2- 172.21.15.59:0/3335620776 >> [v2:172.21.15.176:6800/3291292674,v1:172.21.15.176:6801/3291292674] conn(0x7f2c7006ef80 0x7f2c70071390 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.905+0000 7f2c7affd700 1 -- 172.21.15.59:0/3335620776 <== osd.3 v2:172.21.15.176:6800/3291292674 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f2c70071a40 con 0x7f2c7006ef80 2024-03-19T03:12:24.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.920+0000 7f2c88d95700 1 -- 172.21.15.59:0/3335620776 --> [v2:172.21.15.176:6800/3291292674,v1:172.21.15.176:6801/3291292674] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2c8410ea40 con 0x7f2c7006ef80 2024-03-19T03:12:24.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.920+0000 7f2c7affd700 1 -- 172.21.15.59:0/3335620776 <== osd.3 v2:172.21.15.176:6800/3291292674 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2c8410ea40 con 0x7f2c7006ef80 2024-03-19T03:12:24.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.920+0000 7f2c88d95700 1 -- 172.21.15.59:0/3335620776 >> [v2:172.21.15.176:6800/3291292674,v1:172.21.15.176:6801/3291292674] conn(0x7f2c7006ef80 msgr2=0x7f2c70071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:24.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.920+0000 7f2c88d95700 1 --2- 172.21.15.59:0/3335620776 >> [v2:172.21.15.176:6800/3291292674,v1:172.21.15.176:6801/3291292674] conn(0x7f2c7006ef80 0x7f2c70071390 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:24.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.921+0000 7f2c88d95700 1 -- 172.21.15.59:0/3335620776 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2c700692b0 msgr2=0x7f2c7006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:24.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.921+0000 7f2c88d95700 1 --2- 172.21.15.59:0/3335620776 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2c700692b0 0x7f2c7006b760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2c74004550 tx=0x7f2c74009040 comp rx=0 tx=0).stop 2024-03-19T03:12:24.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.921+0000 7f2c88d95700 1 -- 172.21.15.59:0/3335620776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c840fe830 msgr2=0x7f2c84114180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:24.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.921+0000 7f2c88d95700 1 --2- 172.21.15.59:0/3335620776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c840fe830 0x7f2c84114180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c0093b0 tx=0x7f2c6c009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:24.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.921+0000 7f2c88d95700 1 -- 172.21.15.59:0/3335620776 shutdown_connections 2024-03-19T03:12:24.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.921+0000 7f2c88d95700 1 --2- 172.21.15.59:0/3335620776 >> [v2:172.21.15.176:6800/3291292674,v1:172.21.15.176:6801/3291292674] conn(0x7f2c7006ef80 0x7f2c70071390 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:24.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.921+0000 7f2c88d95700 1 --2- 172.21.15.59:0/3335620776 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2c700692b0 0x7f2c7006b760 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:24.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.921+0000 7f2c88d95700 1 --2- 172.21.15.59:0/3335620776 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c840fde90 0x7f2c84113c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:24.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.921+0000 7f2c88d95700 1 --2- 172.21.15.59:0/3335620776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c840fe830 0x7f2c84114180 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:24.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.921+0000 7f2c88d95700 1 -- 172.21.15.59:0/3335620776 >> 172.21.15.59:0/3335620776 conn(0x7f2c84072bc0 msgr2=0x7f2c84103820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:24.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.921+0000 7f2c88d95700 1 -- 172.21.15.59:0/3335620776 shutdown_connections 2024-03-19T03:12:24.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:24.921+0000 7f2c88d95700 1 -- 172.21.15.59:0/3335620776 wait complete. 2024-03-19T03:12:25.709 INFO:teuthology.orchestra.run.smithi059.stdout:111669149715 2024-03-19T03:12:25.710 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-19T03:12:26.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:26 smithi059 ceph-mon[29133]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:26.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:26 smithi176 ceph-mon[36450]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:26.537 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:26.538 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:26.538 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:26.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.547+0000 7f7e043d1700 1 -- 172.21.15.59:0/3861061602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dfc0fde70 msgr2=0x7f7dfc0fe240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:26.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.547+0000 7f7e043d1700 1 --2- 172.21.15.59:0/3861061602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dfc0fde70 0x7f7dfc0fe240 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7df8004940 tx=0x7f7df8009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:26.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.547+0000 7f7e043d1700 1 -- 172.21.15.59:0/3861061602 shutdown_connections 2024-03-19T03:12:26.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.547+0000 7f7e043d1700 1 --2- 172.21.15.59:0/3861061602 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dfc0fe810 0x7f7dfc106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:26.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.547+0000 7f7e043d1700 1 --2- 172.21.15.59:0/3861061602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dfc0fde70 0x7f7dfc0fe240 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:26.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.547+0000 7f7e043d1700 1 -- 172.21.15.59:0/3861061602 >> 172.21.15.59:0/3861061602 conn(0x7f7dfc0f7880 msgr2=0x7f7dfc0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:26.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.548+0000 7f7e043d1700 1 -- 172.21.15.59:0/3861061602 shutdown_connections 2024-03-19T03:12:26.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.548+0000 7f7e043d1700 1 -- 172.21.15.59:0/3861061602 wait complete. 2024-03-19T03:12:26.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.549+0000 7f7e043d1700 1 Processor -- start 2024-03-19T03:12:26.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.549+0000 7f7e043d1700 1 -- start start 2024-03-19T03:12:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.549+0000 7f7e043d1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dfc0fde70 0x7f7dfc113db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.549+0000 7f7e043d1700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dfc0fe810 0x7f7dfc1142f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.549+0000 7f7e0216d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dfc0fde70 0x7f7dfc113db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.550+0000 7f7e0216d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dfc0fde70 0x7f7dfc113db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36612/0 (socket says 172.21.15.59:36612) 2024-03-19T03:12:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.550+0000 7f7e0216d700 1 -- 172.21.15.59:0/1539752440 learned_addr learned my addr 172.21.15.59:0/1539752440 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.550+0000 7f7e043d1700 1 -- 172.21.15.59:0/1539752440 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dfc114940 con 0x7f7dfc0fde70 2024-03-19T03:12:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.550+0000 7f7e043d1700 1 -- 172.21.15.59:0/1539752440 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dfc10dd10 con 0x7f7dfc0fe810 2024-03-19T03:12:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.550+0000 7f7e0196c700 1 --2- 172.21.15.59:0/1539752440 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dfc0fe810 0x7f7dfc1142f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.550+0000 7f7e0196c700 1 -- 172.21.15.59:0/1539752440 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dfc0fde70 msgr2=0x7f7dfc113db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.550+0000 7f7e0196c700 1 --2- 172.21.15.59:0/1539752440 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dfc0fde70 0x7f7dfc113db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.550+0000 7f7e0196c700 1 -- 172.21.15.59:0/1539752440 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df8005040 con 0x7f7dfc0fe810 2024-03-19T03:12:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.550+0000 7f7e0216d700 1 --2- 172.21.15.59:0/1539752440 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dfc0fde70 0x7f7dfc113db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:12:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.551+0000 7f7e0196c700 1 --2- 172.21.15.59:0/1539752440 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dfc0fe810 0x7f7dfc1142f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7dec009fd0 tx=0x7f7dec00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.551+0000 7f7df2ffd700 1 -- 172.21.15.59:0/1539752440 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7dec004e90 con 0x7f7dfc0fe810 2024-03-19T03:12:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.551+0000 7f7df2ffd700 1 -- 172.21.15.59:0/1539752440 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7dec00f040 con 0x7f7dfc0fe810 2024-03-19T03:12:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.551+0000 7f7df2ffd700 1 -- 172.21.15.59:0/1539752440 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7dec00ca50 con 0x7f7dfc0fe810 2024-03-19T03:12:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.551+0000 7f7e043d1700 1 -- 172.21.15.59:0/1539752440 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dfc10df90 con 0x7f7dfc0fe810 2024-03-19T03:12:26.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.551+0000 7f7e043d1700 1 -- 172.21.15.59:0/1539752440 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dfc10e410 con 0x7f7dfc0fe810 2024-03-19T03:12:26.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.552+0000 7f7e043d1700 1 -- 172.21.15.59:0/1539752440 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7dfc066710 con 0x7f7dfc0fe810 2024-03-19T03:12:26.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.553+0000 7f7df2ffd700 1 -- 172.21.15.59:0/1539752440 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7dec00cbb0 con 0x7f7dfc0fe810 2024-03-19T03:12:26.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.554+0000 7f7df2ffd700 1 --2- 172.21.15.59:0/1539752440 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7de8069370 0x7f7de806b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:26.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.554+0000 7f7e0216d700 1 --2- 172.21.15.59:0/1539752440 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7de8069370 0x7f7de806b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:26.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.554+0000 7f7df2ffd700 1 -- 172.21.15.59:0/1539752440 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7dec086860 con 0x7f7dfc0fe810 2024-03-19T03:12:26.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.554+0000 7f7df2ffd700 1 --2- 172.21.15.59:0/1539752440 >> [v2:172.21.15.176:6816/3279585260,v1:172.21.15.176:6817/3279585260] conn(0x7f7de806eed0 0x7f7de80712e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:26.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.554+0000 7f7df2ffd700 1 -- 172.21.15.59:0/1539752440 --> [v2:172.21.15.176:6816/3279585260,v1:172.21.15.176:6817/3279585260] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7de8071990 con 0x7f7de806eed0 2024-03-19T03:12:26.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.554+0000 7f7df2ffd700 1 -- 172.21.15.59:0/1539752440 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7dec088da0 con 0x7f7dfc0fe810 2024-03-19T03:12:26.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.554+0000 7f7e0216d700 1 --2- 172.21.15.59:0/1539752440 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7de8069370 0x7f7de806b820 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7df8007f50 tx=0x7f7df8009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:26.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.554+0000 7f7e0296e700 1 --2- 172.21.15.59:0/1539752440 >> [v2:172.21.15.176:6816/3279585260,v1:172.21.15.176:6817/3279585260] conn(0x7f7de806eed0 0x7f7de80712e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:26.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.555+0000 7f7e0296e700 1 --2- 172.21.15.59:0/1539752440 >> [v2:172.21.15.176:6816/3279585260,v1:172.21.15.176:6817/3279585260] conn(0x7f7de806eed0 0x7f7de80712e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:26.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.556+0000 7f7df2ffd700 1 -- 172.21.15.59:0/1539752440 <== osd.5 v2:172.21.15.176:6816/3279585260 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f7de8071990 con 0x7f7de806eed0 2024-03-19T03:12:26.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.578+0000 7f7e043d1700 1 -- 172.21.15.59:0/1539752440 --> [v2:172.21.15.176:6816/3279585260,v1:172.21.15.176:6817/3279585260] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7dfc10e840 con 0x7f7de806eed0 2024-03-19T03:12:26.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.579+0000 7f7df2ffd700 1 -- 172.21.15.59:0/1539752440 <== osd.5 v2:172.21.15.176:6816/3279585260 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7dfc10e840 con 0x7f7de806eed0 2024-03-19T03:12:26.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.579+0000 7f7e043d1700 1 -- 172.21.15.59:0/1539752440 >> [v2:172.21.15.176:6816/3279585260,v1:172.21.15.176:6817/3279585260] conn(0x7f7de806eed0 msgr2=0x7f7de80712e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:26.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.579+0000 7f7e043d1700 1 --2- 172.21.15.59:0/1539752440 >> [v2:172.21.15.176:6816/3279585260,v1:172.21.15.176:6817/3279585260] conn(0x7f7de806eed0 0x7f7de80712e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:26.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.580+0000 7f7e043d1700 1 -- 172.21.15.59:0/1539752440 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7de8069370 msgr2=0x7f7de806b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.580+0000 7f7e043d1700 1 --2- 172.21.15.59:0/1539752440 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7de8069370 0x7f7de806b820 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7df8007f50 tx=0x7f7df8009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.580+0000 7f7e043d1700 1 -- 172.21.15.59:0/1539752440 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dfc0fe810 msgr2=0x7f7dfc1142f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.580+0000 7f7e043d1700 1 --2- 172.21.15.59:0/1539752440 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dfc0fe810 0x7f7dfc1142f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7dec009fd0 tx=0x7f7dec00b040 comp rx=0 tx=0).stop 2024-03-19T03:12:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.580+0000 7f7e043d1700 1 -- 172.21.15.59:0/1539752440 shutdown_connections 2024-03-19T03:12:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.580+0000 7f7e043d1700 1 --2- 172.21.15.59:0/1539752440 >> [v2:172.21.15.176:6816/3279585260,v1:172.21.15.176:6817/3279585260] conn(0x7f7de806eed0 0x7f7de80712e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.580+0000 7f7e043d1700 1 --2- 172.21.15.59:0/1539752440 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7de8069370 0x7f7de806b820 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.580+0000 7f7e043d1700 1 --2- 172.21.15.59:0/1539752440 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dfc0fde70 0x7f7dfc113db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.580+0000 7f7e043d1700 1 --2- 172.21.15.59:0/1539752440 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dfc0fe810 0x7f7dfc1142f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.580+0000 7f7e043d1700 1 -- 172.21.15.59:0/1539752440 >> 172.21.15.59:0/1539752440 conn(0x7f7dfc0f7880 msgr2=0x7f7dfc0f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:26.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.580+0000 7f7e043d1700 1 -- 172.21.15.59:0/1539752440 shutdown_connections 2024-03-19T03:12:26.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:26.580+0000 7f7e043d1700 1 -- 172.21.15.59:0/1539752440 wait complete. 2024-03-19T03:12:28.258 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:28 smithi059 ceph-mon[29133]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:28.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:28 smithi176 ceph-mon[36450]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:29.379 INFO:teuthology.orchestra.run.smithi059.stdout:167503724552 2024-03-19T03:12:29.380 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-19T03:12:30.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:30 smithi059 ceph-mon[29133]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:30.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:30 smithi176 ceph-mon[36450]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:30.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.742+0000 7f7b536cd700 1 -- 172.21.15.59:0/2175926024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b4c1037b0 msgr2=0x7f7b4c0fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:30.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.742+0000 7f7b536cd700 1 --2- 172.21.15.59:0/2175926024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b4c1037b0 0x7f7b4c0fce20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7b40004940 tx=0x7f7b40009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:30.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.742+0000 7f7b536cd700 1 -- 172.21.15.59:0/2175926024 shutdown_connections 2024-03-19T03:12:30.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.742+0000 7f7b536cd700 1 --2- 172.21.15.59:0/2175926024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b4c1037b0 0x7f7b4c0fce20 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:30.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.742+0000 7f7b536cd700 1 --2- 172.21.15.59:0/2175926024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b4c102e10 0x7f7b4c1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:30.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.742+0000 7f7b536cd700 1 -- 172.21.15.59:0/2175926024 >> 172.21.15.59:0/2175926024 conn(0x7f7b4c0f89a0 msgr2=0x7f7b4c0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:30.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.742+0000 7f7b536cd700 1 -- 172.21.15.59:0/2175926024 shutdown_connections 2024-03-19T03:12:30.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.742+0000 7f7b536cd700 1 -- 172.21.15.59:0/2175926024 wait complete. 2024-03-19T03:12:30.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.743+0000 7f7b536cd700 1 Processor -- start 2024-03-19T03:12:30.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.743+0000 7f7b536cd700 1 -- start start 2024-03-19T03:12:30.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.743+0000 7f7b536cd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b4c102e10 0x7f7b4c113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:30.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.743+0000 7f7b536cd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b4c1037b0 0x7f7b4c114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:30.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.743+0000 7f7b536cd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c114670 con 0x7f7b4c102e10 2024-03-19T03:12:30.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.743+0000 7f7b536cd700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c10da60 con 0x7f7b4c1037b0 2024-03-19T03:12:30.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.743+0000 7f7b51469700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b4c102e10 0x7f7b4c113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:30.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.743+0000 7f7b51469700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b4c102e10 0x7f7b4c113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36638/0 (socket says 172.21.15.59:36638) 2024-03-19T03:12:30.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.743+0000 7f7b50c68700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b4c1037b0 0x7f7b4c114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:30.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.743+0000 7f7b51469700 1 -- 172.21.15.59:0/1940134807 learned_addr learned my addr 172.21.15.59:0/1940134807 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:30.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.743+0000 7f7b51469700 1 -- 172.21.15.59:0/1940134807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b4c1037b0 msgr2=0x7f7b4c114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:30.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.743+0000 7f7b51469700 1 --2- 172.21.15.59:0/1940134807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b4c1037b0 0x7f7b4c114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:30.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.743+0000 7f7b51469700 1 -- 172.21.15.59:0/1940134807 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b40005040 con 0x7f7b4c102e10 2024-03-19T03:12:30.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.744+0000 7f7b51469700 1 --2- 172.21.15.59:0/1940134807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b4c102e10 0x7f7b4c113ae0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00ec90 tx=0x7f7b3c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:30.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.744+0000 7f7b49ffb700 1 -- 172.21.15.59:0/1940134807 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b3c007b30 con 0x7f7b4c102e10 2024-03-19T03:12:30.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.744+0000 7f7b49ffb700 1 -- 172.21.15.59:0/1940134807 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b3c00f040 con 0x7f7b4c102e10 2024-03-19T03:12:30.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.744+0000 7f7b536cd700 1 -- 172.21.15.59:0/1940134807 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b4c10dd40 con 0x7f7b4c102e10 2024-03-19T03:12:30.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.744+0000 7f7b49ffb700 1 -- 172.21.15.59:0/1940134807 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b3c00cc80 con 0x7f7b4c102e10 2024-03-19T03:12:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.744+0000 7f7b536cd700 1 -- 172.21.15.59:0/1940134807 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b4c10e250 con 0x7f7b4c102e10 2024-03-19T03:12:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.745+0000 7f7b536cd700 1 -- 172.21.15.59:0/1940134807 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7b4c10e750 con 0x7f7b4c102e10 2024-03-19T03:12:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.745+0000 7f7b49ffb700 1 -- 172.21.15.59:0/1940134807 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7b3c007c90 con 0x7f7b4c102e10 2024-03-19T03:12:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.746+0000 7f7b49ffb700 1 --2- 172.21.15.59:0/1940134807 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7b340692a0 0x7f7b3406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.746+0000 7f7b49ffb700 1 -- 172.21.15.59:0/1940134807 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7b3c087220 con 0x7f7b4c102e10 2024-03-19T03:12:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.746+0000 7f7b50c68700 1 --2- 172.21.15.59:0/1940134807 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7b340692a0 0x7f7b3406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.746+0000 7f7b49ffb700 1 --2- 172.21.15.59:0/1940134807 >> [v2:172.21.15.59:6818/639205676,v1:172.21.15.59:6819/639205676] conn(0x7f7b3406ef70 0x7f7b34071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.746+0000 7f7b49ffb700 1 -- 172.21.15.59:0/1940134807 --> [v2:172.21.15.59:6818/639205676,v1:172.21.15.59:6819/639205676] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7b34071a30 con 0x7f7b3406ef70 2024-03-19T03:12:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.746+0000 7f7b49ffb700 1 -- 172.21.15.59:0/1940134807 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b3c089870 con 0x7f7b4c102e10 2024-03-19T03:12:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.746+0000 7f7b51c6a700 1 --2- 172.21.15.59:0/1940134807 >> [v2:172.21.15.59:6818/639205676,v1:172.21.15.59:6819/639205676] conn(0x7f7b3406ef70 0x7f7b34071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.746+0000 7f7b50c68700 1 --2- 172.21.15.59:0/1940134807 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7b340692a0 0x7f7b3406b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7b40004890 tx=0x7f7b40009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.747+0000 7f7b51c6a700 1 --2- 172.21.15.59:0/1940134807 >> [v2:172.21.15.59:6818/639205676,v1:172.21.15.59:6819/639205676] conn(0x7f7b3406ef70 0x7f7b34071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:30.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.747+0000 7f7b49ffb700 1 -- 172.21.15.59:0/1940134807 <== osd.2 v2:172.21.15.59:6818/639205676 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f7b34071a30 con 0x7f7b3406ef70 2024-03-19T03:12:30.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.762+0000 7f7b536cd700 1 -- 172.21.15.59:0/1940134807 --> [v2:172.21.15.59:6818/639205676,v1:172.21.15.59:6819/639205676] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7b4c05e6c0 con 0x7f7b3406ef70 2024-03-19T03:12:30.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.763+0000 7f7b49ffb700 1 -- 172.21.15.59:0/1940134807 <== osd.2 v2:172.21.15.59:6818/639205676 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7b4c05e6c0 con 0x7f7b3406ef70 2024-03-19T03:12:30.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.763+0000 7f7b536cd700 1 -- 172.21.15.59:0/1940134807 >> [v2:172.21.15.59:6818/639205676,v1:172.21.15.59:6819/639205676] conn(0x7f7b3406ef70 msgr2=0x7f7b34071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:30.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.763+0000 7f7b536cd700 1 --2- 172.21.15.59:0/1940134807 >> [v2:172.21.15.59:6818/639205676,v1:172.21.15.59:6819/639205676] conn(0x7f7b3406ef70 0x7f7b34071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:30.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.763+0000 7f7b536cd700 1 -- 172.21.15.59:0/1940134807 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7b340692a0 msgr2=0x7f7b3406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:30.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.763+0000 7f7b536cd700 1 --2- 172.21.15.59:0/1940134807 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7b340692a0 0x7f7b3406b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7b40004890 tx=0x7f7b40009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:30.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.763+0000 7f7b536cd700 1 -- 172.21.15.59:0/1940134807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b4c102e10 msgr2=0x7f7b4c113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:30.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.763+0000 7f7b536cd700 1 --2- 172.21.15.59:0/1940134807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b4c102e10 0x7f7b4c113ae0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00ec90 tx=0x7f7b3c00b040 comp rx=0 tx=0).stop 2024-03-19T03:12:30.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.763+0000 7f7b536cd700 1 -- 172.21.15.59:0/1940134807 shutdown_connections 2024-03-19T03:12:30.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.763+0000 7f7b536cd700 1 --2- 172.21.15.59:0/1940134807 >> [v2:172.21.15.59:6818/639205676,v1:172.21.15.59:6819/639205676] conn(0x7f7b3406ef70 0x7f7b34071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:30.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.763+0000 7f7b536cd700 1 --2- 172.21.15.59:0/1940134807 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7b340692a0 0x7f7b3406b750 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:30.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.763+0000 7f7b536cd700 1 --2- 172.21.15.59:0/1940134807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b4c102e10 0x7f7b4c113ae0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:30.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.763+0000 7f7b536cd700 1 --2- 172.21.15.59:0/1940134807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b4c1037b0 0x7f7b4c114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:30.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.763+0000 7f7b536cd700 1 -- 172.21.15.59:0/1940134807 >> 172.21.15.59:0/1940134807 conn(0x7f7b4c0f89a0 msgr2=0x7f7b4c1021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:30.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.763+0000 7f7b536cd700 1 -- 172.21.15.59:0/1940134807 shutdown_connections 2024-03-19T03:12:30.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:30.763+0000 7f7b536cd700 1 -- 172.21.15.59:0/1940134807 wait complete. 2024-03-19T03:12:31.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:31 smithi059 ceph-mon[29133]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:31.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:12:31.548 INFO:teuthology.orchestra.run.smithi059.stdout:81604378649 2024-03-19T03:12:31.549 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-19T03:12:31.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:31 smithi176 ceph-mon[36450]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:31.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:31 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:12:32.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.102+0000 7f624ae3b700 1 -- 172.21.15.59:0/932496051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6244102e00 msgr2=0x7f62441031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:32.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.102+0000 7f624ae3b700 1 --2- 172.21.15.59:0/932496051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6244102e00 0x7f62441031d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6238004940 tx=0x7f6238009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:32.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.102+0000 7f624ae3b700 1 -- 172.21.15.59:0/932496051 shutdown_connections 2024-03-19T03:12:32.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.102+0000 7f624ae3b700 1 --2- 172.21.15.59:0/932496051 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62441037a0 0x7f62440fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:32.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.102+0000 7f624ae3b700 1 --2- 172.21.15.59:0/932496051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6244102e00 0x7f62441031d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:32.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.102+0000 7f624ae3b700 1 -- 172.21.15.59:0/932496051 >> 172.21.15.59:0/932496051 conn(0x7f62440f89b0 msgr2=0x7f62440fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:32.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.102+0000 7f624ae3b700 1 -- 172.21.15.59:0/932496051 shutdown_connections 2024-03-19T03:12:32.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.102+0000 7f624ae3b700 1 -- 172.21.15.59:0/932496051 wait complete. 2024-03-19T03:12:32.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.103+0000 7f624ae3b700 1 Processor -- start 2024-03-19T03:12:32.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.103+0000 7f624ae3b700 1 -- start start 2024-03-19T03:12:32.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.103+0000 7f624ae3b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6244102e00 0x7f6244115b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:32.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.103+0000 7f624ae3b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62441037a0 0x7f62441160a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:32.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.103+0000 7f624ae3b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6244116780 con 0x7f6244102e00 2024-03-19T03:12:32.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.103+0000 7f624ae3b700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624410fcd0 con 0x7f62441037a0 2024-03-19T03:12:32.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.103+0000 7f6248bd7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6244102e00 0x7f6244115b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:32.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.103+0000 7f6248bd7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6244102e00 0x7f6244115b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36650/0 (socket says 172.21.15.59:36650) 2024-03-19T03:12:32.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.103+0000 7f6248bd7700 1 -- 172.21.15.59:0/3329116109 learned_addr learned my addr 172.21.15.59:0/3329116109 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:32.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.104+0000 7f6243fff700 1 --2- 172.21.15.59:0/3329116109 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62441037a0 0x7f62441160a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:32.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.104+0000 7f6248bd7700 1 -- 172.21.15.59:0/3329116109 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62441037a0 msgr2=0x7f62441160a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:32.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.104+0000 7f6248bd7700 1 --2- 172.21.15.59:0/3329116109 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62441037a0 0x7f62441160a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:32.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.104+0000 7f6248bd7700 1 -- 172.21.15.59:0/3329116109 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6238005040 con 0x7f6244102e00 2024-03-19T03:12:32.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.104+0000 7f6243fff700 1 --2- 172.21.15.59:0/3329116109 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62441037a0 0x7f62441160a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T03:12:32.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.104+0000 7f6248bd7700 1 --2- 172.21.15.59:0/3329116109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6244102e00 0x7f6244115b60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f623800b3b0 tx=0x7f6238009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:32.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.104+0000 7f62417fa700 1 -- 172.21.15.59:0/3329116109 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f623800e040 con 0x7f6244102e00 2024-03-19T03:12:32.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.104+0000 7f624ae3b700 1 -- 172.21.15.59:0/3329116109 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f624410fef0 con 0x7f6244102e00 2024-03-19T03:12:32.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.104+0000 7f62417fa700 1 -- 172.21.15.59:0/3329116109 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f623800bbc0 con 0x7f6244102e00 2024-03-19T03:12:32.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.104+0000 7f624ae3b700 1 -- 172.21.15.59:0/3329116109 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62441104b0 con 0x7f6244102e00 2024-03-19T03:12:32.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.104+0000 7f62417fa700 1 -- 172.21.15.59:0/3329116109 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f623801c3f0 con 0x7f6244102e00 2024-03-19T03:12:32.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.105+0000 7f624ae3b700 1 -- 172.21.15.59:0/3329116109 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6244110aa0 con 0x7f6244102e00 2024-03-19T03:12:32.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.106+0000 7f62417fa700 1 -- 172.21.15.59:0/3329116109 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f623800bd30 con 0x7f6244102e00 2024-03-19T03:12:32.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.106+0000 7f62417fa700 1 --2- 172.21.15.59:0/3329116109 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f62340692a0 0x7f623406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:32.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.106+0000 7f62417fa700 1 -- 172.21.15.59:0/3329116109 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6238088010 con 0x7f6244102e00 2024-03-19T03:12:32.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.106+0000 7f6243fff700 1 --2- 172.21.15.59:0/3329116109 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f62340692a0 0x7f623406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:32.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.106+0000 7f62417fa700 1 --2- 172.21.15.59:0/3329116109 >> [v2:172.21.15.59:6810/3595398218,v1:172.21.15.59:6811/3595398218] conn(0x7f623406ef70 0x7f6234071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:32.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.106+0000 7f62417fa700 1 -- 172.21.15.59:0/3329116109 --> [v2:172.21.15.59:6810/3595398218,v1:172.21.15.59:6811/3595398218] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6234071a30 con 0x7f623406ef70 2024-03-19T03:12:32.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.106+0000 7f62417fa700 1 -- 172.21.15.59:0/3329116109 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f623808a600 con 0x7f6244102e00 2024-03-19T03:12:32.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.106+0000 7f62493d8700 1 --2- 172.21.15.59:0/3329116109 >> [v2:172.21.15.59:6810/3595398218,v1:172.21.15.59:6811/3595398218] conn(0x7f623406ef70 0x7f6234071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:32.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.106+0000 7f6243fff700 1 --2- 172.21.15.59:0/3329116109 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f62340692a0 0x7f623406b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6230006fd0 tx=0x7f6230008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:32.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.107+0000 7f62493d8700 1 --2- 172.21.15.59:0/3329116109 >> [v2:172.21.15.59:6810/3595398218,v1:172.21.15.59:6811/3595398218] conn(0x7f623406ef70 0x7f6234071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:32.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.107+0000 7f62417fa700 1 -- 172.21.15.59:0/3329116109 <== osd.1 v2:172.21.15.59:6810/3595398218 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f6234071a30 con 0x7f623406ef70 2024-03-19T03:12:32.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.122+0000 7f624ae3b700 1 -- 172.21.15.59:0/3329116109 --> [v2:172.21.15.59:6810/3595398218,v1:172.21.15.59:6811/3595398218] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6244110080 con 0x7f623406ef70 2024-03-19T03:12:32.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.122+0000 7f62417fa700 1 -- 172.21.15.59:0/3329116109 <== osd.1 v2:172.21.15.59:6810/3595398218 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6244110080 con 0x7f623406ef70 2024-03-19T03:12:32.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.123+0000 7f624ae3b700 1 -- 172.21.15.59:0/3329116109 >> [v2:172.21.15.59:6810/3595398218,v1:172.21.15.59:6811/3595398218] conn(0x7f623406ef70 msgr2=0x7f6234071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:32.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.123+0000 7f624ae3b700 1 --2- 172.21.15.59:0/3329116109 >> [v2:172.21.15.59:6810/3595398218,v1:172.21.15.59:6811/3595398218] conn(0x7f623406ef70 0x7f6234071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:32.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.123+0000 7f624ae3b700 1 -- 172.21.15.59:0/3329116109 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f62340692a0 msgr2=0x7f623406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:32.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.123+0000 7f624ae3b700 1 --2- 172.21.15.59:0/3329116109 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f62340692a0 0x7f623406b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6230006fd0 tx=0x7f6230008040 comp rx=0 tx=0).stop 2024-03-19T03:12:32.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.123+0000 7f624ae3b700 1 -- 172.21.15.59:0/3329116109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6244102e00 msgr2=0x7f6244115b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:32.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.123+0000 7f624ae3b700 1 --2- 172.21.15.59:0/3329116109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6244102e00 0x7f6244115b60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f623800b3b0 tx=0x7f6238009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:32.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.123+0000 7f624ae3b700 1 -- 172.21.15.59:0/3329116109 shutdown_connections 2024-03-19T03:12:32.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.123+0000 7f624ae3b700 1 --2- 172.21.15.59:0/3329116109 >> [v2:172.21.15.59:6810/3595398218,v1:172.21.15.59:6811/3595398218] conn(0x7f623406ef70 0x7f6234071380 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:32.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.123+0000 7f624ae3b700 1 --2- 172.21.15.59:0/3329116109 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f62340692a0 0x7f623406b750 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:32.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.123+0000 7f624ae3b700 1 --2- 172.21.15.59:0/3329116109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6244102e00 0x7f6244115b60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:32.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.123+0000 7f624ae3b700 1 --2- 172.21.15.59:0/3329116109 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62441037a0 0x7f62441160a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:32.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.123+0000 7f624ae3b700 1 -- 172.21.15.59:0/3329116109 >> 172.21.15.59:0/3329116109 conn(0x7f62440f89b0 msgr2=0x7f6244071c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:32.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.123+0000 7f624ae3b700 1 -- 172.21.15.59:0/3329116109 shutdown_connections 2024-03-19T03:12:32.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:32.123+0000 7f624ae3b700 1 -- 172.21.15.59:0/3329116109 wait complete. 2024-03-19T03:12:33.392 INFO:teuthology.orchestra.run.smithi059.stdout:60129542175 2024-03-19T03:12:33.393 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-19T03:12:33.998 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:33.998 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:34.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.028+0000 7f9453e44700 1 -- 172.21.15.59:0/1884004004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f944c1026b0 msgr2=0x7f944c0fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:34.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.028+0000 7f9453e44700 1 --2- 172.21.15.59:0/1884004004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f944c1026b0 0x7f944c0fbd20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f943c004940 tx=0x7f943c009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:34.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.028+0000 7f9453e44700 1 -- 172.21.15.59:0/1884004004 shutdown_connections 2024-03-19T03:12:34.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.028+0000 7f9453e44700 1 --2- 172.21.15.59:0/1884004004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f944c1026b0 0x7f944c0fbd20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:34.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.028+0000 7f9453e44700 1 --2- 172.21.15.59:0/1884004004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f944c101d10 0x7f944c1020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:34.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.028+0000 7f9453e44700 1 -- 172.21.15.59:0/1884004004 >> 172.21.15.59:0/1884004004 conn(0x7f944c0f7880 msgr2=0x7f944c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:34.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.029+0000 7f9453e44700 1 -- 172.21.15.59:0/1884004004 shutdown_connections 2024-03-19T03:12:34.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.029+0000 7f9453e44700 1 -- 172.21.15.59:0/1884004004 wait complete. 2024-03-19T03:12:34.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.030+0000 7f9453e44700 1 Processor -- start 2024-03-19T03:12:34.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.030+0000 7f9453e44700 1 -- start start 2024-03-19T03:12:34.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.030+0000 7f9453e44700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f944c101d10 0x7f944c112e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:34.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.030+0000 7f9453e44700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f944c1026b0 0x7f944c113340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:34.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.030+0000 7f9453e44700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944c113990 con 0x7f944c101d10 2024-03-19T03:12:34.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.030+0000 7f9453e44700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944c10cd60 con 0x7f944c1026b0 2024-03-19T03:12:34.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.030+0000 7f9451be0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f944c101d10 0x7f944c112e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:34.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.030+0000 7f9451be0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f944c101d10 0x7f944c112e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37112/0 (socket says 172.21.15.59:37112) 2024-03-19T03:12:34.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.030+0000 7f9451be0700 1 -- 172.21.15.59:0/170750943 learned_addr learned my addr 172.21.15.59:0/170750943 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:34.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.030+0000 7f94513df700 1 --2- 172.21.15.59:0/170750943 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f944c1026b0 0x7f944c113340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:34.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.031+0000 7f9451be0700 1 -- 172.21.15.59:0/170750943 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f944c1026b0 msgr2=0x7f944c113340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:34.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.031+0000 7f9451be0700 1 --2- 172.21.15.59:0/170750943 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f944c1026b0 0x7f944c113340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:34.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.031+0000 7f9451be0700 1 -- 172.21.15.59:0/170750943 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f943c005040 con 0x7f944c101d10 2024-03-19T03:12:34.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.031+0000 7f94513df700 1 --2- 172.21.15.59:0/170750943 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f944c1026b0 0x7f944c113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T03:12:34.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.031+0000 7f9451be0700 1 --2- 172.21.15.59:0/170750943 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f944c101d10 0x7f944c112e00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f944800fce0 tx=0x7f94480046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:34.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.031+0000 7f94427fc700 1 -- 172.21.15.59:0/170750943 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f944800fe90 con 0x7f944c101d10 2024-03-19T03:12:34.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.031+0000 7f9453e44700 1 -- 172.21.15.59:0/170750943 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f944c10cfe0 con 0x7f944c101d10 2024-03-19T03:12:34.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.031+0000 7f94427fc700 1 -- 172.21.15.59:0/170750943 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f944800fe90 con 0x7f944c101d10 2024-03-19T03:12:34.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.031+0000 7f9453e44700 1 -- 172.21.15.59:0/170750943 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f944c10d5a0 con 0x7f944c101d10 2024-03-19T03:12:34.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.031+0000 7f94427fc700 1 -- 172.21.15.59:0/170750943 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9448009980 con 0x7f944c101d10 2024-03-19T03:12:34.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.032+0000 7f9453e44700 1 -- 172.21.15.59:0/170750943 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f944c0fd460 con 0x7f944c101d10 2024-03-19T03:12:34.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.033+0000 7f94427fc700 1 -- 172.21.15.59:0/170750943 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f9448007ab0 con 0x7f944c101d10 2024-03-19T03:12:34.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.033+0000 7f94427fc700 1 --2- 172.21.15.59:0/170750943 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f9438069310 0x7f943806b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:34.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.033+0000 7f94513df700 1 --2- 172.21.15.59:0/170750943 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f9438069310 0x7f943806b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:34.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.033+0000 7f94427fc700 1 -- 172.21.15.59:0/170750943 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9448088fc0 con 0x7f944c101d10 2024-03-19T03:12:34.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.033+0000 7f94427fc700 1 --2- 172.21.15.59:0/170750943 >> [v2:172.21.15.176:6808/1006246702,v1:172.21.15.176:6809/1006246702] conn(0x7f943806efe0 0x7f94380713f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:34.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.033+0000 7f94427fc700 1 -- 172.21.15.59:0/170750943 --> [v2:172.21.15.176:6808/1006246702,v1:172.21.15.176:6809/1006246702] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9438071aa0 con 0x7f943806efe0 2024-03-19T03:12:34.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.033+0000 7f94427fc700 1 -- 172.21.15.59:0/170750943 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f944808b550 con 0x7f944c101d10 2024-03-19T03:12:34.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.034+0000 7f94513df700 1 --2- 172.21.15.59:0/170750943 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f9438069310 0x7f943806b7c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f943c009fd0 tx=0x7f943c009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:34.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.034+0000 7f94523e1700 1 --2- 172.21.15.59:0/170750943 >> [v2:172.21.15.176:6808/1006246702,v1:172.21.15.176:6809/1006246702] conn(0x7f943806efe0 0x7f94380713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:34.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.034+0000 7f94523e1700 1 --2- 172.21.15.59:0/170750943 >> [v2:172.21.15.176:6808/1006246702,v1:172.21.15.176:6809/1006246702] conn(0x7f943806efe0 0x7f94380713f0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:34.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.035+0000 7f94427fc700 1 -- 172.21.15.59:0/170750943 <== osd.4 v2:172.21.15.176:6808/1006246702 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f9438071aa0 con 0x7f943806efe0 2024-03-19T03:12:34.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.050+0000 7f9453e44700 1 -- 172.21.15.59:0/170750943 --> [v2:172.21.15.176:6808/1006246702,v1:172.21.15.176:6809/1006246702] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f944c10d170 con 0x7f943806efe0 2024-03-19T03:12:34.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.051+0000 7f94427fc700 1 -- 172.21.15.59:0/170750943 <== osd.4 v2:172.21.15.176:6808/1006246702 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f944c10d170 con 0x7f943806efe0 2024-03-19T03:12:34.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.051+0000 7f9453e44700 1 -- 172.21.15.59:0/170750943 >> [v2:172.21.15.176:6808/1006246702,v1:172.21.15.176:6809/1006246702] conn(0x7f943806efe0 msgr2=0x7f94380713f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:34.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.051+0000 7f9453e44700 1 --2- 172.21.15.59:0/170750943 >> [v2:172.21.15.176:6808/1006246702,v1:172.21.15.176:6809/1006246702] conn(0x7f943806efe0 0x7f94380713f0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:34.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.051+0000 7f9453e44700 1 -- 172.21.15.59:0/170750943 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f9438069310 msgr2=0x7f943806b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:34.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.051+0000 7f9453e44700 1 --2- 172.21.15.59:0/170750943 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f9438069310 0x7f943806b7c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f943c009fd0 tx=0x7f943c009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:34.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.051+0000 7f9453e44700 1 -- 172.21.15.59:0/170750943 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f944c101d10 msgr2=0x7f944c112e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:34.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.051+0000 7f9453e44700 1 --2- 172.21.15.59:0/170750943 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f944c101d10 0x7f944c112e00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f944800fce0 tx=0x7f94480046b0 comp rx=0 tx=0).stop 2024-03-19T03:12:34.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.051+0000 7f9453e44700 1 -- 172.21.15.59:0/170750943 shutdown_connections 2024-03-19T03:12:34.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.051+0000 7f9453e44700 1 --2- 172.21.15.59:0/170750943 >> [v2:172.21.15.176:6808/1006246702,v1:172.21.15.176:6809/1006246702] conn(0x7f943806efe0 0x7f94380713f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:34.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.051+0000 7f9453e44700 1 --2- 172.21.15.59:0/170750943 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f9438069310 0x7f943806b7c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:34.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.051+0000 7f9453e44700 1 --2- 172.21.15.59:0/170750943 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f944c101d10 0x7f944c112e00 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:34.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.051+0000 7f9453e44700 1 --2- 172.21.15.59:0/170750943 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f944c1026b0 0x7f944c113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:34.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.051+0000 7f9453e44700 1 -- 172.21.15.59:0/170750943 >> 172.21.15.59:0/170750943 conn(0x7f944c0f7880 msgr2=0x7f944c0f8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:34.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.052+0000 7f9453e44700 1 -- 172.21.15.59:0/170750943 shutdown_connections 2024-03-19T03:12:34.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.052+0000 7f9453e44700 1 -- 172.21.15.59:0/170750943 wait complete. 2024-03-19T03:12:34.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.247+0000 7f2a32a7e700 1 -- 172.21.15.59:0/1505272491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a2c101d70 msgr2=0x7f2a2c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:34.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.247+0000 7f2a32a7e700 1 --2- 172.21.15.59:0/1505272491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a2c101d70 0x7f2a2c102140 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2a24009ce0 tx=0x7f2a24004d10 comp rx=0 tx=0).stop 2024-03-19T03:12:34.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.247+0000 7f2a32a7e700 1 -- 172.21.15.59:0/1505272491 shutdown_connections 2024-03-19T03:12:34.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.247+0000 7f2a32a7e700 1 --2- 172.21.15.59:0/1505272491 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a2c102710 0x7f2a2c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:34.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.247+0000 7f2a32a7e700 1 --2- 172.21.15.59:0/1505272491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a2c101d70 0x7f2a2c102140 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:34.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.247+0000 7f2a32a7e700 1 -- 172.21.15.59:0/1505272491 >> 172.21.15.59:0/1505272491 conn(0x7f2a2c072bc0 msgr2=0x7f2a2c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:34.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.247+0000 7f2a32a7e700 1 -- 172.21.15.59:0/1505272491 shutdown_connections 2024-03-19T03:12:34.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.247+0000 7f2a32a7e700 1 -- 172.21.15.59:0/1505272491 wait complete. 2024-03-19T03:12:34.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.247+0000 7f2a32a7e700 1 Processor -- start 2024-03-19T03:12:34.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.247+0000 7f2a32a7e700 1 -- start start 2024-03-19T03:12:34.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.248+0000 7f2a32a7e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a2c101d70 0x7f2a2c10ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:34.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.248+0000 7f2a32a7e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a2c102710 0x7f2a2c10f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:34.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.248+0000 7f2a32a7e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c10f850 con 0x7f2a2c101d70 2024-03-19T03:12:34.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.248+0000 7f2a32a7e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c10f990 con 0x7f2a2c102710 2024-03-19T03:12:34.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.248+0000 7f2a3081a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a2c101d70 0x7f2a2c10ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:34.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.248+0000 7f2a3081a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a2c101d70 0x7f2a2c10ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37132/0 (socket says 172.21.15.59:37132) 2024-03-19T03:12:34.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.248+0000 7f2a3081a700 1 -- 172.21.15.59:0/2141732756 learned_addr learned my addr 172.21.15.59:0/2141732756 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:34.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.248+0000 7f2a2bfff700 1 --2- 172.21.15.59:0/2141732756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a2c102710 0x7f2a2c10f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:34.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.248+0000 7f2a3081a700 1 -- 172.21.15.59:0/2141732756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a2c102710 msgr2=0x7f2a2c10f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:34.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.248+0000 7f2a3081a700 1 --2- 172.21.15.59:0/2141732756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a2c102710 0x7f2a2c10f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:34.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.248+0000 7f2a3081a700 1 -- 172.21.15.59:0/2141732756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a24009940 con 0x7f2a2c101d70 2024-03-19T03:12:34.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.248+0000 7f2a3081a700 1 --2- 172.21.15.59:0/2141732756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a2c101d70 0x7f2a2c10ec30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2a2400be90 tx=0x7f2a24004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:34.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.248+0000 7f2a297fa700 1 -- 172.21.15.59:0/2141732756 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a2400e040 con 0x7f2a2c101d70 2024-03-19T03:12:34.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.248+0000 7f2a32a7e700 1 -- 172.21.15.59:0/2141732756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a2c117ca0 con 0x7f2a2c101d70 2024-03-19T03:12:34.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.248+0000 7f2a297fa700 1 -- 172.21.15.59:0/2141732756 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a24004610 con 0x7f2a2c101d70 2024-03-19T03:12:34.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.248+0000 7f2a297fa700 1 -- 172.21.15.59:0/2141732756 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a2401c6d0 con 0x7f2a2c101d70 2024-03-19T03:12:34.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.249+0000 7f2a32a7e700 1 -- 172.21.15.59:0/2141732756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a2c118230 con 0x7f2a2c101d70 2024-03-19T03:12:34.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.249+0000 7f2a32a7e700 1 -- 172.21.15.59:0/2141732756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2a2c0fd4c0 con 0x7f2a2c101d70 2024-03-19T03:12:34.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.250+0000 7f2a297fa700 1 -- 172.21.15.59:0/2141732756 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f2a24012b00 con 0x7f2a2c101d70 2024-03-19T03:12:34.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.250+0000 7f2a297fa700 1 --2- 172.21.15.59:0/2141732756 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2a180692a0 0x7f2a1806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:34.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.250+0000 7f2a297fa700 1 -- 172.21.15.59:0/2141732756 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2a24088cd0 con 0x7f2a2c101d70 2024-03-19T03:12:34.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.250+0000 7f2a2bfff700 1 --2- 172.21.15.59:0/2141732756 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2a180692a0 0x7f2a1806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:34.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.251+0000 7f2a297fa700 1 --2- 172.21.15.59:0/2141732756 >> [v2:172.21.15.59:6802/1770897334,v1:172.21.15.59:6803/1770897334] conn(0x7f2a1806ef70 0x7f2a18071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:34.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.251+0000 7f2a297fa700 1 -- 172.21.15.59:0/2141732756 --> [v2:172.21.15.59:6802/1770897334,v1:172.21.15.59:6803/1770897334] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a18071a30 con 0x7f2a1806ef70 2024-03-19T03:12:34.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.251+0000 7f2a297fa700 1 -- 172.21.15.59:0/2141732756 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a240890b0 con 0x7f2a2c101d70 2024-03-19T03:12:34.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.251+0000 7f2a3101b700 1 --2- 172.21.15.59:0/2141732756 >> [v2:172.21.15.59:6802/1770897334,v1:172.21.15.59:6803/1770897334] conn(0x7f2a1806ef70 0x7f2a18071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:34.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.251+0000 7f2a2bfff700 1 --2- 172.21.15.59:0/2141732756 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2a180692a0 0x7f2a1806b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c066000 tx=0x7f2a1c0091a0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:34.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.251+0000 7f2a3101b700 1 --2- 172.21.15.59:0/2141732756 >> [v2:172.21.15.59:6802/1770897334,v1:172.21.15.59:6803/1770897334] conn(0x7f2a1806ef70 0x7f2a18071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:34.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.252+0000 7f2a297fa700 1 -- 172.21.15.59:0/2141732756 <== osd.0 v2:172.21.15.59:6802/1770897334 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f2a18071a30 con 0x7f2a1806ef70 2024-03-19T03:12:34.255 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:34 smithi059 ceph-mon[29133]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:34.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.267+0000 7f2a32a7e700 1 -- 172.21.15.59:0/2141732756 --> [v2:172.21.15.59:6802/1770897334,v1:172.21.15.59:6803/1770897334] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2a2c117e30 con 0x7f2a1806ef70 2024-03-19T03:12:34.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.268+0000 7f2a297fa700 1 -- 172.21.15.59:0/2141732756 <== osd.0 v2:172.21.15.59:6802/1770897334 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2a2c117e30 con 0x7f2a1806ef70 2024-03-19T03:12:34.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.268+0000 7f2a32a7e700 1 -- 172.21.15.59:0/2141732756 >> [v2:172.21.15.59:6802/1770897334,v1:172.21.15.59:6803/1770897334] conn(0x7f2a1806ef70 msgr2=0x7f2a18071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:34.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.268+0000 7f2a32a7e700 1 --2- 172.21.15.59:0/2141732756 >> [v2:172.21.15.59:6802/1770897334,v1:172.21.15.59:6803/1770897334] conn(0x7f2a1806ef70 0x7f2a18071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:34.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.268+0000 7f2a32a7e700 1 -- 172.21.15.59:0/2141732756 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2a180692a0 msgr2=0x7f2a1806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:34.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.268+0000 7f2a32a7e700 1 --2- 172.21.15.59:0/2141732756 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2a180692a0 0x7f2a1806b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c066000 tx=0x7f2a1c0091a0 comp rx=0 tx=0).stop 2024-03-19T03:12:34.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.268+0000 7f2a32a7e700 1 -- 172.21.15.59:0/2141732756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a2c101d70 msgr2=0x7f2a2c10ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:34.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.268+0000 7f2a32a7e700 1 --2- 172.21.15.59:0/2141732756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a2c101d70 0x7f2a2c10ec30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2a2400be90 tx=0x7f2a24004d10 comp rx=0 tx=0).stop 2024-03-19T03:12:34.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.268+0000 7f2a32a7e700 1 -- 172.21.15.59:0/2141732756 shutdown_connections 2024-03-19T03:12:34.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.268+0000 7f2a32a7e700 1 --2- 172.21.15.59:0/2141732756 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2a180692a0 0x7f2a1806b750 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:34.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.268+0000 7f2a32a7e700 1 --2- 172.21.15.59:0/2141732756 >> [v2:172.21.15.59:6802/1770897334,v1:172.21.15.59:6803/1770897334] conn(0x7f2a1806ef70 0x7f2a18071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:34.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.268+0000 7f2a32a7e700 1 --2- 172.21.15.59:0/2141732756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a2c101d70 0x7f2a2c10ec30 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:34.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.268+0000 7f2a32a7e700 1 --2- 172.21.15.59:0/2141732756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a2c102710 0x7f2a2c10f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:34.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.268+0000 7f2a32a7e700 1 -- 172.21.15.59:0/2141732756 >> 172.21.15.59:0/2141732756 conn(0x7f2a2c072bc0 msgr2=0x7f2a2c0fae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:34.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.269+0000 7f2a32a7e700 1 -- 172.21.15.59:0/2141732756 shutdown_connections 2024-03-19T03:12:34.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:34.269+0000 7f2a32a7e700 1 -- 172.21.15.59:0/2141732756 wait complete. 2024-03-19T03:12:34.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:34 smithi176 ceph-mon[36450]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:35.681 INFO:teuthology.orchestra.run.smithi059.stdout:38654705701 2024-03-19T03:12:35.681 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-19T03:12:36.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:36 smithi059 ceph-mon[29133]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:36.176 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:36.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:36 smithi176 ceph-mon[36450]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:37.202 INFO:teuthology.orchestra.run.smithi059.stdout:146028888079 2024-03-19T03:12:37.202 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-19T03:12:37.749 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:38.275 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:38 smithi059 ceph-mon[29133]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:38.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:38 smithi176 ceph-mon[36450]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:39.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.105+0000 7fe7a6acd700 1 -- 172.21.15.59:0/2689575407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a0102710 msgr2=0x7fe7a00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:39.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.105+0000 7fe7a6acd700 1 --2- 172.21.15.59:0/2689575407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a0102710 0x7fe7a00fbd80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe790004940 tx=0x7fe790009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:39.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.105+0000 7fe7a6acd700 1 -- 172.21.15.59:0/2689575407 shutdown_connections 2024-03-19T03:12:39.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.105+0000 7fe7a6acd700 1 --2- 172.21.15.59:0/2689575407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a0102710 0x7fe7a00fbd80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:39.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.105+0000 7fe7a6acd700 1 --2- 172.21.15.59:0/2689575407 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7a0101d70 0x7fe7a0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:39.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.105+0000 7fe7a6acd700 1 -- 172.21.15.59:0/2689575407 >> 172.21.15.59:0/2689575407 conn(0x7fe7a0072bc0 msgr2=0x7fe7a0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:39.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.105+0000 7fe7a6acd700 1 -- 172.21.15.59:0/2689575407 shutdown_connections 2024-03-19T03:12:39.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.105+0000 7fe7a6acd700 1 -- 172.21.15.59:0/2689575407 wait complete. 2024-03-19T03:12:39.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.105+0000 7fe7a6acd700 1 Processor -- start 2024-03-19T03:12:39.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.105+0000 7fe7a6acd700 1 -- start start 2024-03-19T03:12:39.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.106+0000 7fe7a6acd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a0101d70 0x7fe7a010eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:39.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.106+0000 7fe7a6acd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7a0102710 0x7fe7a010f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:39.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.106+0000 7fe7a6acd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a010f720 con 0x7fe7a0101d70 2024-03-19T03:12:39.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.106+0000 7fe7a6acd700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a010f860 con 0x7fe7a0102710 2024-03-19T03:12:39.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.106+0000 7fe7a4869700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a0101d70 0x7fe7a010eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:39.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.106+0000 7fe7a4869700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a0101d70 0x7fe7a010eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37146/0 (socket says 172.21.15.59:37146) 2024-03-19T03:12:39.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.106+0000 7fe7a4869700 1 -- 172.21.15.59:0/2441608938 learned_addr learned my addr 172.21.15.59:0/2441608938 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:39.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.106+0000 7fe79ffff700 1 --2- 172.21.15.59:0/2441608938 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7a0102710 0x7fe7a010f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:39.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.106+0000 7fe7a4869700 1 -- 172.21.15.59:0/2441608938 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7a0102710 msgr2=0x7fe7a010f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:39.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.106+0000 7fe7a4869700 1 --2- 172.21.15.59:0/2441608938 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7a0102710 0x7fe7a010f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:39.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.106+0000 7fe7a4869700 1 -- 172.21.15.59:0/2441608938 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe790005040 con 0x7fe7a0101d70 2024-03-19T03:12:39.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.106+0000 7fe79ffff700 1 --2- 172.21.15.59:0/2441608938 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7a0102710 0x7fe7a010f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:12:39.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.106+0000 7fe7a4869700 1 --2- 172.21.15.59:0/2441608938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a0101d70 0x7fe7a010eca0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe79800cac0 tx=0x7fe798007ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:39.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.107+0000 7fe79d7fa700 1 -- 172.21.15.59:0/2441608938 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe79800aa50 con 0x7fe7a0101d70 2024-03-19T03:12:39.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.107+0000 7fe79d7fa700 1 -- 172.21.15.59:0/2441608938 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe79800abb0 con 0x7fe7a0101d70 2024-03-19T03:12:39.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.107+0000 7fe7a6acd700 1 -- 172.21.15.59:0/2441608938 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7a0117b50 con 0x7fe7a0101d70 2024-03-19T03:12:39.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.107+0000 7fe79d7fa700 1 -- 172.21.15.59:0/2441608938 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe7980115e0 con 0x7fe7a0101d70 2024-03-19T03:12:39.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.107+0000 7fe7a6acd700 1 -- 172.21.15.59:0/2441608938 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7a0118110 con 0x7fe7a0101d70 2024-03-19T03:12:39.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.108+0000 7fe7a6acd700 1 -- 172.21.15.59:0/2441608938 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7a0104240 con 0x7fe7a0101d70 2024-03-19T03:12:39.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.108+0000 7fe79d7fa700 1 -- 172.21.15.59:0/2441608938 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe798012440 con 0x7fe7a0101d70 2024-03-19T03:12:39.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.108+0000 7fe79d7fa700 1 --2- 172.21.15.59:0/2441608938 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe78c0692a0 0x7fe78c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:39.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.108+0000 7fe79d7fa700 1 -- 172.21.15.59:0/2441608938 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe798087560 con 0x7fe7a0101d70 2024-03-19T03:12:39.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.108+0000 7fe79ffff700 1 --2- 172.21.15.59:0/2441608938 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe78c0692a0 0x7fe78c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:39.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.109+0000 7fe79ffff700 1 --2- 172.21.15.59:0/2441608938 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe78c0692a0 0x7fe78c06b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe79000b3b0 tx=0x7fe790009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:39.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.113+0000 7fe79d7fa700 1 -- 172.21.15.59:0/2441608938 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fe798053bd0 con 0x7fe7a0101d70 2024-03-19T03:12:39.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.317+0000 7fe7a6acd700 1 -- 172.21.15.59:0/2441608938 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fe7a00fd4c0 con 0x7fe7a0101d70 2024-03-19T03:12:39.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.317+0000 7fe79d7fa700 1 -- 172.21.15.59:0/2441608938 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe7980571f0 con 0x7fe7a0101d70 2024-03-19T03:12:39.319 INFO:teuthology.orchestra.run.smithi059.stdout:81604378650 2024-03-19T03:12:39.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.319+0000 7fe7a6acd700 1 -- 172.21.15.59:0/2441608938 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe78c0692a0 msgr2=0x7fe78c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:39.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.319+0000 7fe7a6acd700 1 --2- 172.21.15.59:0/2441608938 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe78c0692a0 0x7fe78c06b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe79000b3b0 tx=0x7fe790009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:39.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.319+0000 7fe7a6acd700 1 -- 172.21.15.59:0/2441608938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a0101d70 msgr2=0x7fe7a010eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:39.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.319+0000 7fe7a6acd700 1 --2- 172.21.15.59:0/2441608938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a0101d70 0x7fe7a010eca0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe79800cac0 tx=0x7fe798007ee0 comp rx=0 tx=0).stop 2024-03-19T03:12:39.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.319+0000 7fe7a6acd700 1 -- 172.21.15.59:0/2441608938 shutdown_connections 2024-03-19T03:12:39.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.319+0000 7fe7a6acd700 1 --2- 172.21.15.59:0/2441608938 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe78c0692a0 0x7fe78c06b750 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:39.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.319+0000 7fe7a6acd700 1 --2- 172.21.15.59:0/2441608938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a0101d70 0x7fe7a010eca0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:39.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.319+0000 7fe7a6acd700 1 --2- 172.21.15.59:0/2441608938 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7a0102710 0x7fe7a010f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:39.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.319+0000 7fe7a6acd700 1 -- 172.21.15.59:0/2441608938 >> 172.21.15.59:0/2441608938 conn(0x7fe7a0072bc0 msgr2=0x7fe7a00fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:39.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.319+0000 7fe7a6acd700 1 -- 172.21.15.59:0/2441608938 shutdown_connections 2024-03-19T03:12:39.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:39.320+0000 7fe7a6acd700 1 -- 172.21.15.59:0/2441608938 wait complete. 2024-03-19T03:12:39.910 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378649 got 81604378650 for osd.2 2024-03-19T03:12:39.910 DEBUG:teuthology.parallel:result is None 2024-03-19T03:12:40.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.087+0000 7ff816c07700 1 -- 172.21.15.59:0/2835051609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8100fb4a0 msgr2=0x7ff8100fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:40.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.087+0000 7ff816c07700 1 --2- 172.21.15.59:0/2835051609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8100fb4a0 0x7ff8100fb910 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8004940 tx=0x7ff7f8009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:40.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.087+0000 7ff816c07700 1 -- 172.21.15.59:0/2835051609 shutdown_connections 2024-03-19T03:12:40.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.087+0000 7ff816c07700 1 --2- 172.21.15.59:0/2835051609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8100fb4a0 0x7ff8100fb910 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:40.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.087+0000 7ff816c07700 1 --2- 172.21.15.59:0/2835051609 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff810102e70 0x7ff810103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:40.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.087+0000 7ff816c07700 1 -- 172.21.15.59:0/2835051609 >> 172.21.15.59:0/2835051609 conn(0x7ff8100714c0 msgr2=0x7ff8100718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:40.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.087+0000 7ff816c07700 1 -- 172.21.15.59:0/2835051609 shutdown_connections 2024-03-19T03:12:40.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.087+0000 7ff816c07700 1 -- 172.21.15.59:0/2835051609 wait complete. 2024-03-19T03:12:40.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.088+0000 7ff816c07700 1 Processor -- start 2024-03-19T03:12:40.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.088+0000 7ff816c07700 1 -- start start 2024-03-19T03:12:40.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.088+0000 7ff816c07700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8100fb4a0 0x7ff8100fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:40.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.088+0000 7ff816c07700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff810102e70 0x7ff8100fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:40.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.088+0000 7ff816c07700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8100fd150 con 0x7ff8100fb4a0 2024-03-19T03:12:40.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.088+0000 7ff816c07700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8100fd290 con 0x7ff810102e70 2024-03-19T03:12:40.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.088+0000 7ff8149a3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8100fb4a0 0x7ff8100fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:40.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.088+0000 7ff8149a3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8100fb4a0 0x7ff8100fc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37156/0 (socket says 172.21.15.59:37156) 2024-03-19T03:12:40.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.089+0000 7ff8149a3700 1 -- 172.21.15.59:0/1250524549 learned_addr learned my addr 172.21.15.59:0/1250524549 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:40.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.089+0000 7ff80ffff700 1 --2- 172.21.15.59:0/1250524549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff810102e70 0x7ff8100fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:40.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.089+0000 7ff8149a3700 1 -- 172.21.15.59:0/1250524549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff810102e70 msgr2=0x7ff8100fcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:40.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.089+0000 7ff8149a3700 1 --2- 172.21.15.59:0/1250524549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff810102e70 0x7ff8100fcbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:40.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.089+0000 7ff8149a3700 1 -- 172.21.15.59:0/1250524549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f8005040 con 0x7ff8100fb4a0 2024-03-19T03:12:40.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.089+0000 7ff8149a3700 1 --2- 172.21.15.59:0/1250524549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8100fb4a0 0x7ff8100fc680 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff80000c8c0 tx=0x7ff8000044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:40.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.089+0000 7ff80d7fa700 1 -- 172.21.15.59:0/1250524549 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff80000a760 con 0x7ff8100fb4a0 2024-03-19T03:12:40.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.089+0000 7ff816c07700 1 -- 172.21.15.59:0/1250524549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff810119e70 con 0x7ff8100fb4a0 2024-03-19T03:12:40.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.089+0000 7ff80d7fa700 1 -- 172.21.15.59:0/1250524549 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff80000ad90 con 0x7ff8100fb4a0 2024-03-19T03:12:40.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.089+0000 7ff816c07700 1 -- 172.21.15.59:0/1250524549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff81011a2f0 con 0x7ff8100fb4a0 2024-03-19T03:12:40.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.090+0000 7ff80d7fa700 1 -- 172.21.15.59:0/1250524549 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8000113a0 con 0x7ff8100fb4a0 2024-03-19T03:12:40.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.091+0000 7ff816c07700 1 -- 172.21.15.59:0/1250524549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff810100110 con 0x7ff8100fb4a0 2024-03-19T03:12:40.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.091+0000 7ff80d7fa700 1 -- 172.21.15.59:0/1250524549 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff800012400 con 0x7ff8100fb4a0 2024-03-19T03:12:40.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.092+0000 7ff80d7fa700 1 --2- 172.21.15.59:0/1250524549 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7ff7fc0692a0 0x7ff7fc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:40.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.092+0000 7ff80d7fa700 1 -- 172.21.15.59:0/1250524549 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff800086eb0 con 0x7ff8100fb4a0 2024-03-19T03:12:40.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.092+0000 7ff80ffff700 1 --2- 172.21.15.59:0/1250524549 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7ff7fc0692a0 0x7ff7fc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:40.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.092+0000 7ff80ffff700 1 --2- 172.21.15.59:0/1250524549 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7ff7fc0692a0 0x7ff7fc06b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8004f90 tx=0x7ff7f8009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:40.097 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:40.097 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:40.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.098+0000 7ff80d7fa700 1 -- 172.21.15.59:0/1250524549 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7ff800053520 con 0x7ff8100fb4a0 2024-03-19T03:12:40.100 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:40 smithi059 ceph-mon[29133]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:40.100 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:40 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/2441608938' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-19T03:12:40.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.299+0000 7ff816c07700 1 -- 172.21.15.59:0/1250524549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7ff810109f70 con 0x7ff8100fb4a0 2024-03-19T03:12:40.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.299+0000 7ff80d7fa700 1 -- 172.21.15.59:0/1250524549 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff800056b40 con 0x7ff8100fb4a0 2024-03-19T03:12:40.301 INFO:teuthology.orchestra.run.smithi059.stdout:60129542176 2024-03-19T03:12:40.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.301+0000 7ff816c07700 1 -- 172.21.15.59:0/1250524549 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7ff7fc0692a0 msgr2=0x7ff7fc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:40.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.301+0000 7ff816c07700 1 --2- 172.21.15.59:0/1250524549 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7ff7fc0692a0 0x7ff7fc06b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8004f90 tx=0x7ff7f8009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:40.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.301+0000 7ff816c07700 1 -- 172.21.15.59:0/1250524549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8100fb4a0 msgr2=0x7ff8100fc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:40.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.301+0000 7ff816c07700 1 --2- 172.21.15.59:0/1250524549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8100fb4a0 0x7ff8100fc680 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff80000c8c0 tx=0x7ff8000044c0 comp rx=0 tx=0).stop 2024-03-19T03:12:40.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.301+0000 7ff816c07700 1 -- 172.21.15.59:0/1250524549 shutdown_connections 2024-03-19T03:12:40.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.301+0000 7ff816c07700 1 --2- 172.21.15.59:0/1250524549 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7ff7fc0692a0 0x7ff7fc06b750 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:40.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.301+0000 7ff816c07700 1 --2- 172.21.15.59:0/1250524549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8100fb4a0 0x7ff8100fc680 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:40.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.301+0000 7ff816c07700 1 --2- 172.21.15.59:0/1250524549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff810102e70 0x7ff8100fcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:40.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.301+0000 7ff816c07700 1 -- 172.21.15.59:0/1250524549 >> 172.21.15.59:0/1250524549 conn(0x7ff8100714c0 msgr2=0x7ff8100fa810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:40.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.301+0000 7ff816c07700 1 -- 172.21.15.59:0/1250524549 shutdown_connections 2024-03-19T03:12:40.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:40.302+0000 7ff816c07700 1 -- 172.21.15.59:0/1250524549 wait complete. 2024-03-19T03:12:40.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:40 smithi176 ceph-mon[36450]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:40.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:40 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/2441608938' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-19T03:12:41.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:41 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/1250524549' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-19T03:12:41.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:41 smithi059 ceph-mon[29133]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:41.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:41 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/1250524549' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-19T03:12:41.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:41 smithi176 ceph-mon[36450]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:43.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.211+0000 7f448a61f700 1 -- 172.21.15.59:0/1467176446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44840fb3b0 msgr2=0x7f44840fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:43.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.211+0000 7f448a61f700 1 --2- 172.21.15.59:0/1467176446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44840fb3b0 0x7f44840fb820 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4478004940 tx=0x7f4478009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:43.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.211+0000 7f448a61f700 1 -- 172.21.15.59:0/1467176446 shutdown_connections 2024-03-19T03:12:43.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.211+0000 7f448a61f700 1 --2- 172.21.15.59:0/1467176446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44840fb3b0 0x7f44840fb820 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:43.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.211+0000 7f448a61f700 1 --2- 172.21.15.59:0/1467176446 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4484102f70 0x7f4484103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:43.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.211+0000 7f448a61f700 1 -- 172.21.15.59:0/1467176446 >> 172.21.15.59:0/1467176446 conn(0x7f4484072310 msgr2=0x7f4484072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:43.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.212+0000 7f448a61f700 1 -- 172.21.15.59:0/1467176446 shutdown_connections 2024-03-19T03:12:43.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.212+0000 7f448a61f700 1 -- 172.21.15.59:0/1467176446 wait complete. 2024-03-19T03:12:43.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.212+0000 7f448a61f700 1 Processor -- start 2024-03-19T03:12:43.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.212+0000 7f448a61f700 1 -- start start 2024-03-19T03:12:43.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.212+0000 7f448a61f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44840fb3b0 0x7f44841139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:43.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.212+0000 7f448a61f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4484102f70 0x7f4484113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:43.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.212+0000 7f448a61f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44841145c0 con 0x7f44840fb3b0 2024-03-19T03:12:43.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.212+0000 7f448a61f700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448410db50 con 0x7f4484102f70 2024-03-19T03:12:43.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.212+0000 7f4483fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44840fb3b0 0x7f44841139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:43.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.212+0000 7f4483fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44840fb3b0 0x7f44841139a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37182/0 (socket says 172.21.15.59:37182) 2024-03-19T03:12:43.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.212+0000 7f4483fff700 1 -- 172.21.15.59:0/3523463497 learned_addr learned my addr 172.21.15.59:0/3523463497 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:43.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.213+0000 7f44837fe700 1 --2- 172.21.15.59:0/3523463497 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4484102f70 0x7f4484113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:43.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.213+0000 7f4483fff700 1 -- 172.21.15.59:0/3523463497 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4484102f70 msgr2=0x7f4484113ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:43.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.213+0000 7f4483fff700 1 --2- 172.21.15.59:0/3523463497 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4484102f70 0x7f4484113ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:43.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.213+0000 7f4483fff700 1 -- 172.21.15.59:0/3523463497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4478005040 con 0x7f44840fb3b0 2024-03-19T03:12:43.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.213+0000 7f4483fff700 1 --2- 172.21.15.59:0/3523463497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44840fb3b0 0x7f44841139a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f447400b790 tx=0x7f4474009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:43.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.213+0000 7f4480ff9700 1 -- 172.21.15.59:0/3523463497 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f447400f6f0 con 0x7f44840fb3b0 2024-03-19T03:12:43.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.213+0000 7f448a61f700 1 -- 172.21.15.59:0/3523463497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f448410de00 con 0x7f44840fb3b0 2024-03-19T03:12:43.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.213+0000 7f4480ff9700 1 -- 172.21.15.59:0/3523463497 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f447400fd20 con 0x7f44840fb3b0 2024-03-19T03:12:43.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.213+0000 7f4480ff9700 1 -- 172.21.15.59:0/3523463497 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44740103a0 con 0x7f44840fb3b0 2024-03-19T03:12:43.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.213+0000 7f448a61f700 1 -- 172.21.15.59:0/3523463497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f448410e310 con 0x7f44840fb3b0 2024-03-19T03:12:43.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.214+0000 7f448a61f700 1 -- 172.21.15.59:0/3523463497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44840fe6d0 con 0x7f44840fb3b0 2024-03-19T03:12:43.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.215+0000 7f4480ff9700 1 -- 172.21.15.59:0/3523463497 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4474011400 con 0x7f44840fb3b0 2024-03-19T03:12:43.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.215+0000 7f4480ff9700 1 --2- 172.21.15.59:0/3523463497 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f446c0692a0 0x7f446c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:43.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.215+0000 7f4480ff9700 1 -- 172.21.15.59:0/3523463497 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4474086c30 con 0x7f44840fb3b0 2024-03-19T03:12:43.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.215+0000 7f44837fe700 1 --2- 172.21.15.59:0/3523463497 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f446c0692a0 0x7f446c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:43.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.215+0000 7f44837fe700 1 --2- 172.21.15.59:0/3523463497 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f446c0692a0 0x7f446c06b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f448410f2b0 tx=0x7f4478009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:43.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.220+0000 7f4480ff9700 1 -- 172.21.15.59:0/3523463497 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f44740532a0 con 0x7f44840fb3b0 2024-03-19T03:12:43.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.419+0000 7f448a61f700 1 -- 172.21.15.59:0/3523463497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f448410eff0 con 0x7f44840fb3b0 2024-03-19T03:12:43.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.419+0000 7f4480ff9700 1 -- 172.21.15.59:0/3523463497 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f44740568c0 con 0x7f44840fb3b0 2024-03-19T03:12:43.421 INFO:teuthology.orchestra.run.smithi059.stdout:111669149719 2024-03-19T03:12:43.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.421+0000 7f448a61f700 1 -- 172.21.15.59:0/3523463497 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f446c0692a0 msgr2=0x7f446c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:43.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.421+0000 7f448a61f700 1 --2- 172.21.15.59:0/3523463497 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f446c0692a0 0x7f446c06b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f448410f2b0 tx=0x7f4478009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:43.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.421+0000 7f448a61f700 1 -- 172.21.15.59:0/3523463497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44840fb3b0 msgr2=0x7f44841139a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:43.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.421+0000 7f448a61f700 1 --2- 172.21.15.59:0/3523463497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44840fb3b0 0x7f44841139a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f447400b790 tx=0x7f4474009250 comp rx=0 tx=0).stop 2024-03-19T03:12:43.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.421+0000 7f448a61f700 1 -- 172.21.15.59:0/3523463497 shutdown_connections 2024-03-19T03:12:43.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.421+0000 7f448a61f700 1 --2- 172.21.15.59:0/3523463497 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f446c0692a0 0x7f446c06b750 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:43.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.421+0000 7f448a61f700 1 --2- 172.21.15.59:0/3523463497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44840fb3b0 0x7f44841139a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:43.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.421+0000 7f448a61f700 1 --2- 172.21.15.59:0/3523463497 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4484102f70 0x7f4484113ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:43.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.421+0000 7f448a61f700 1 -- 172.21.15.59:0/3523463497 >> 172.21.15.59:0/3523463497 conn(0x7f4484072310 msgr2=0x7f44840f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:43.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.422+0000 7f448a61f700 1 -- 172.21.15.59:0/3523463497 shutdown_connections 2024-03-19T03:12:43.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.422+0000 7f448a61f700 1 -- 172.21.15.59:0/3523463497 wait complete. 2024-03-19T03:12:43.870 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542175 got 60129542176 for osd.1 2024-03-19T03:12:43.870 DEBUG:teuthology.parallel:result is None 2024-03-19T03:12:44.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:44 smithi059 ceph-mon[29133]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:44.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:44 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/3523463497' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-19T03:12:44.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:44 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/1351771275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-19T03:12:44.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:44 smithi176 ceph-mon[36450]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:44.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:44 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/3523463497' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-19T03:12:44.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:44 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/1351771275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-19T03:12:44.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.479+0000 7fb4d4f97700 1 -- 172.21.15.59:0/2218543173 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4d00fe830 msgr2=0x7fb4d0106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:44.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.479+0000 7fb4d4f97700 1 --2- 172.21.15.59:0/2218543173 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4d00fe830 0x7fb4d0106f00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4004940 tx=0x7fb4c4009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:44.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.479+0000 7fb4d4f97700 1 -- 172.21.15.59:0/2218543173 shutdown_connections 2024-03-19T03:12:44.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.479+0000 7fb4d4f97700 1 --2- 172.21.15.59:0/2218543173 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4d00fe830 0x7fb4d0106f00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:44.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.479+0000 7fb4d4f97700 1 --2- 172.21.15.59:0/2218543173 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb4d00fde90 0x7fb4d00fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:44.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.479+0000 7fb4d4f97700 1 -- 172.21.15.59:0/2218543173 >> 172.21.15.59:0/2218543173 conn(0x7fb4d0072bc0 msgr2=0x7fb4d0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:44.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.479+0000 7fb4d4f97700 1 -- 172.21.15.59:0/2218543173 shutdown_connections 2024-03-19T03:12:44.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.479+0000 7fb4d4f97700 1 -- 172.21.15.59:0/2218543173 wait complete. 2024-03-19T03:12:44.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.480+0000 7fb4d4f97700 1 Processor -- start 2024-03-19T03:12:44.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.480+0000 7fb4d4f97700 1 -- start start 2024-03-19T03:12:44.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.480+0000 7fb4d4f97700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4d00fde90 0x7fb4d00fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:44.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.480+0000 7fb4d4f97700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb4d00fe830 0x7fb4d00fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:44.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.480+0000 7fb4d4f97700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d00fc8b0 con 0x7fb4d00fde90 2024-03-19T03:12:44.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.480+0000 7fb4d4f97700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d00fc9f0 con 0x7fb4d00fe830 2024-03-19T03:12:44.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.480+0000 7fb4ce59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4d00fde90 0x7fb4d00fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:44.718 INFO:teuthology.orchestra.run.smithi059.stdout:167503724555 2024-03-19T03:12:44.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.480+0000 7fb4ce59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4d00fde90 0x7fb4d00fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37204/0 (socket says 172.21.15.59:37204) 2024-03-19T03:12:44.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.480+0000 7fb4ce59c700 1 -- 172.21.15.59:0/1351771275 learned_addr learned my addr 172.21.15.59:0/1351771275 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:44.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.481+0000 7fb4cdd9b700 1 --2- 172.21.15.59:0/1351771275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb4d00fe830 0x7fb4d00fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:44.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.481+0000 7fb4ce59c700 1 -- 172.21.15.59:0/1351771275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb4d00fe830 msgr2=0x7fb4d00fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:44.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.481+0000 7fb4ce59c700 1 --2- 172.21.15.59:0/1351771275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb4d00fe830 0x7fb4d00fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:44.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.481+0000 7fb4ce59c700 1 -- 172.21.15.59:0/1351771275 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c4005040 con 0x7fb4d00fde90 2024-03-19T03:12:44.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.481+0000 7fb4ce59c700 1 --2- 172.21.15.59:0/1351771275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4d00fde90 0x7fb4d00fbd50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb4c000b790 tx=0x7fb4c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:44.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.481+0000 7fb4beffd700 1 -- 172.21.15.59:0/1351771275 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4c000f770 con 0x7fb4d00fde90 2024-03-19T03:12:44.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.481+0000 7fb4d4f97700 1 -- 172.21.15.59:0/1351771275 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4d0119e90 con 0x7fb4d00fde90 2024-03-19T03:12:44.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.481+0000 7fb4beffd700 1 -- 172.21.15.59:0/1351771275 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb4c000fda0 con 0x7fb4d00fde90 2024-03-19T03:12:44.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.481+0000 7fb4beffd700 1 -- 172.21.15.59:0/1351771275 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4c000d310 con 0x7fb4d00fde90 2024-03-19T03:12:44.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.481+0000 7fb4d4f97700 1 -- 172.21.15.59:0/1351771275 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d011a290 con 0x7fb4d00fde90 2024-03-19T03:12:44.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.482+0000 7fb4d4f97700 1 -- 172.21.15.59:0/1351771275 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4d0065e30 con 0x7fb4d00fde90 2024-03-19T03:12:44.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.482+0000 7fb4beffd700 1 -- 172.21.15.59:0/1351771275 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb4c0010420 con 0x7fb4d00fde90 2024-03-19T03:12:44.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.483+0000 7fb4beffd700 1 --2- 172.21.15.59:0/1351771275 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb4b80692a0 0x7fb4b806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:44.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.483+0000 7fb4beffd700 1 -- 172.21.15.59:0/1351771275 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb4c0086e40 con 0x7fb4d00fde90 2024-03-19T03:12:44.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.483+0000 7fb4cdd9b700 1 --2- 172.21.15.59:0/1351771275 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb4b80692a0 0x7fb4b806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:44.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.483+0000 7fb4cdd9b700 1 --2- 172.21.15.59:0/1351771275 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb4b80692a0 0x7fb4b806b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb4d00fd690 tx=0x7fb4c4009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:44.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.488+0000 7fb4beffd700 1 -- 172.21.15.59:0/1351771275 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fb4c00534b0 con 0x7fb4d00fde90 2024-03-19T03:12:44.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.679+0000 7fb4d4f97700 1 -- 172.21.15.59:0/1351771275 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fb4d005f5f0 con 0x7fb4d00fde90 2024-03-19T03:12:44.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.679+0000 7fb4beffd700 1 -- 172.21.15.59:0/1351771275 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb4c0056ad0 con 0x7fb4d00fde90 2024-03-19T03:12:44.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.681+0000 7fb4d4f97700 1 -- 172.21.15.59:0/1351771275 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb4b80692a0 msgr2=0x7fb4b806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:44.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.681+0000 7fb4d4f97700 1 --2- 172.21.15.59:0/1351771275 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb4b80692a0 0x7fb4b806b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb4d00fd690 tx=0x7fb4c4009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:44.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.681+0000 7fb4d4f97700 1 -- 172.21.15.59:0/1351771275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4d00fde90 msgr2=0x7fb4d00fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:44.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.681+0000 7fb4d4f97700 1 --2- 172.21.15.59:0/1351771275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4d00fde90 0x7fb4d00fbd50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb4c000b790 tx=0x7fb4c0009250 comp rx=0 tx=0).stop 2024-03-19T03:12:44.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.681+0000 7fb4d4f97700 1 -- 172.21.15.59:0/1351771275 shutdown_connections 2024-03-19T03:12:44.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.681+0000 7fb4d4f97700 1 --2- 172.21.15.59:0/1351771275 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb4b80692a0 0x7fb4b806b750 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:44.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.681+0000 7fb4d4f97700 1 --2- 172.21.15.59:0/1351771275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4d00fde90 0x7fb4d00fbd50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:44.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.681+0000 7fb4d4f97700 1 --2- 172.21.15.59:0/1351771275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb4d00fe830 0x7fb4d00fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:44.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.681+0000 7fb4d4f97700 1 -- 172.21.15.59:0/1351771275 >> 172.21.15.59:0/1351771275 conn(0x7fb4d0072bc0 msgr2=0x7fb4d0103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:44.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.681+0000 7fb4d4f97700 1 -- 172.21.15.59:0/1351771275 shutdown_connections 2024-03-19T03:12:44.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:43.681+0000 7fb4d4f97700 1 -- 172.21.15.59:0/1351771275 wait complete. 2024-03-19T03:12:45.492 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724552 got 167503724555 for osd.5 2024-03-19T03:12:45.494 DEBUG:teuthology.parallel:result is None 2024-03-19T03:12:46.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:46 smithi059 ceph-mon[29133]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:46.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:12:46.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:46 smithi176 ceph-mon[36450]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:46.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:46 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:12:46.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.511+0000 7f2abf691700 1 -- 172.21.15.59:0/1138564142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ab81037a0 msgr2=0x7f2ab80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:46.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.511+0000 7f2abf691700 1 --2- 172.21.15.59:0/1138564142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ab81037a0 0x7f2ab80fce10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8004940 tx=0x7f2aa8009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:46.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.511+0000 7f2abf691700 1 -- 172.21.15.59:0/1138564142 shutdown_connections 2024-03-19T03:12:46.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.511+0000 7f2abf691700 1 --2- 172.21.15.59:0/1138564142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ab81037a0 0x7f2ab80fce10 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:46.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.511+0000 7f2abf691700 1 --2- 172.21.15.59:0/1138564142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ab8102e00 0x7f2ab81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:46.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.511+0000 7f2abf691700 1 -- 172.21.15.59:0/1138564142 >> 172.21.15.59:0/1138564142 conn(0x7f2ab80f89b0 msgr2=0x7f2ab80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:46.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.511+0000 7f2abf691700 1 -- 172.21.15.59:0/1138564142 shutdown_connections 2024-03-19T03:12:46.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.511+0000 7f2abf691700 1 -- 172.21.15.59:0/1138564142 wait complete. 2024-03-19T03:12:46.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.512+0000 7f2abf691700 1 Processor -- start 2024-03-19T03:12:46.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.512+0000 7f2abf691700 1 -- start start 2024-03-19T03:12:46.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.512+0000 7f2abf691700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ab8102e00 0x7f2ab80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:46.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.512+0000 7f2abf691700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ab81037a0 0x7f2ab8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:46.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.512+0000 7f2abf691700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab80722a0 con 0x7f2ab81037a0 2024-03-19T03:12:46.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.512+0000 7f2abf691700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab8070d40 con 0x7f2ab8102e00 2024-03-19T03:12:46.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.512+0000 7f2abcc2c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ab81037a0 0x7f2ab8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:46.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.512+0000 7f2abcc2c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ab81037a0 0x7f2ab8070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32902/0 (socket says 172.21.15.59:32902) 2024-03-19T03:12:46.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.512+0000 7f2abcc2c700 1 -- 172.21.15.59:0/94409866 learned_addr learned my addr 172.21.15.59:0/94409866 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:46.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.512+0000 7f2abd42d700 1 --2- 172.21.15.59:0/94409866 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ab8102e00 0x7f2ab80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:46.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.512+0000 7f2abcc2c700 1 -- 172.21.15.59:0/94409866 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ab8102e00 msgr2=0x7f2ab80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:46.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.512+0000 7f2abcc2c700 1 --2- 172.21.15.59:0/94409866 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ab8102e00 0x7f2ab80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:46.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.513+0000 7f2abcc2c700 1 -- 172.21.15.59:0/94409866 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa8005040 con 0x7f2ab81037a0 2024-03-19T03:12:46.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.513+0000 7f2abd42d700 1 --2- 172.21.15.59:0/94409866 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ab8102e00 0x7f2ab80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:12:46.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.513+0000 7f2abcc2c700 1 --2- 172.21.15.59:0/94409866 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ab81037a0 0x7f2ab8070800 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8009100 tx=0x7f2aa8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:46.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.513+0000 7f2aadffb700 1 -- 172.21.15.59:0/94409866 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2aa800e040 con 0x7f2ab81037a0 2024-03-19T03:12:46.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.513+0000 7f2aadffb700 1 -- 172.21.15.59:0/94409866 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2aa8004290 con 0x7f2ab81037a0 2024-03-19T03:12:46.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.513+0000 7f2aadffb700 1 -- 172.21.15.59:0/94409866 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2aa801c3f0 con 0x7f2ab81037a0 2024-03-19T03:12:46.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.513+0000 7f2abf691700 1 -- 172.21.15.59:0/94409866 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab8070fc0 con 0x7f2ab81037a0 2024-03-19T03:12:46.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.513+0000 7f2abf691700 1 -- 172.21.15.59:0/94409866 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab811a1d0 con 0x7f2ab81037a0 2024-03-19T03:12:46.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.514+0000 7f2abf691700 1 -- 172.21.15.59:0/94409866 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab8100110 con 0x7f2ab81037a0 2024-03-19T03:12:46.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.514+0000 7f2aadffb700 1 -- 172.21.15.59:0/94409866 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f2aa8013790 con 0x7f2ab81037a0 2024-03-19T03:12:46.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.515+0000 7f2aadffb700 1 --2- 172.21.15.59:0/94409866 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2aa40692a0 0x7f2aa406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:46.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.515+0000 7f2aadffb700 1 -- 172.21.15.59:0/94409866 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2aa8088140 con 0x7f2ab81037a0 2024-03-19T03:12:46.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.515+0000 7f2abd42d700 1 --2- 172.21.15.59:0/94409866 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2aa40692a0 0x7f2aa406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:46.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.515+0000 7f2abd42d700 1 --2- 172.21.15.59:0/94409866 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2aa40692a0 0x7f2aa406b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4009b10 tx=0x7f2ab4008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:46.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.520+0000 7f2aadffb700 1 -- 172.21.15.59:0/94409866 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f2aa80547b0 con 0x7f2ab81037a0 2024-03-19T03:12:46.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.717+0000 7f2abf691700 1 -- 172.21.15.59:0/94409866 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f2ab80fe550 con 0x7f2ab81037a0 2024-03-19T03:12:46.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.717+0000 7f2aadffb700 1 -- 172.21.15.59:0/94409866 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2aa8013420 con 0x7f2ab81037a0 2024-03-19T03:12:46.719 INFO:teuthology.orchestra.run.smithi059.stdout:38654705703 2024-03-19T03:12:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.719+0000 7f2abf691700 1 -- 172.21.15.59:0/94409866 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2aa40692a0 msgr2=0x7f2aa406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.719+0000 7f2abf691700 1 --2- 172.21.15.59:0/94409866 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2aa40692a0 0x7f2aa406b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4009b10 tx=0x7f2ab4008040 comp rx=0 tx=0).stop 2024-03-19T03:12:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.719+0000 7f2abf691700 1 -- 172.21.15.59:0/94409866 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ab81037a0 msgr2=0x7f2ab8070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.719+0000 7f2abf691700 1 --2- 172.21.15.59:0/94409866 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ab81037a0 0x7f2ab8070800 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8009100 tx=0x7f2aa8009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.719+0000 7f2abf691700 1 -- 172.21.15.59:0/94409866 shutdown_connections 2024-03-19T03:12:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.719+0000 7f2abf691700 1 --2- 172.21.15.59:0/94409866 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2aa40692a0 0x7f2aa406b750 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.719+0000 7f2abf691700 1 --2- 172.21.15.59:0/94409866 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ab8102e00 0x7f2ab80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.719+0000 7f2abf691700 1 --2- 172.21.15.59:0/94409866 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ab81037a0 0x7f2ab8070800 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.719+0000 7f2abf691700 1 -- 172.21.15.59:0/94409866 >> 172.21.15.59:0/94409866 conn(0x7f2ab80f89b0 msgr2=0x7f2ab80fa770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:46.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.719+0000 7f2abf691700 1 -- 172.21.15.59:0/94409866 shutdown_connections 2024-03-19T03:12:46.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:46.719+0000 7f2abf691700 1 -- 172.21.15.59:0/94409866 wait complete. 2024-03-19T03:12:47.121 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149715 got 111669149719 for osd.3 2024-03-19T03:12:47.121 DEBUG:teuthology.parallel:result is None 2024-03-19T03:12:47.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:47 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/94409866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-19T03:12:47.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:47 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/94409866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-19T03:12:47.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.671+0000 7f64161b5700 1 -- 172.21.15.59:0/2395933461 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64100fe830 msgr2=0x7f6410106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:47.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.671+0000 7f64161b5700 1 --2- 172.21.15.59:0/2395933461 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64100fe830 0x7f6410106f00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f63f8004940 tx=0x7f63f8009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:47.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.671+0000 7f64161b5700 1 -- 172.21.15.59:0/2395933461 shutdown_connections 2024-03-19T03:12:47.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.671+0000 7f64161b5700 1 --2- 172.21.15.59:0/2395933461 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64100fe830 0x7f6410106f00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:47.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.671+0000 7f64161b5700 1 --2- 172.21.15.59:0/2395933461 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64100fde90 0x7f64100fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:47.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.671+0000 7f64161b5700 1 -- 172.21.15.59:0/2395933461 >> 172.21.15.59:0/2395933461 conn(0x7f6410072bc0 msgr2=0x7f6410072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:47.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.671+0000 7f64161b5700 1 -- 172.21.15.59:0/2395933461 shutdown_connections 2024-03-19T03:12:47.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.671+0000 7f64161b5700 1 -- 172.21.15.59:0/2395933461 wait complete. 2024-03-19T03:12:47.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.671+0000 7f64161b5700 1 Processor -- start 2024-03-19T03:12:47.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.672+0000 7f64161b5700 1 -- start start 2024-03-19T03:12:47.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.672+0000 7f64161b5700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64100fde90 0x7f6410115db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:47.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.672+0000 7f64161b5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64100fe830 0x7f64101162f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:47.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.672+0000 7f64161b5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6410116940 con 0x7f64100fe830 2024-03-19T03:12:47.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.672+0000 7f64161b5700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641010fd10 con 0x7f64100fde90 2024-03-19T03:12:47.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.672+0000 7f640effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64100fe830 0x7f64101162f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:47.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.672+0000 7f640effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64100fe830 0x7f64101162f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32916/0 (socket says 172.21.15.59:32916) 2024-03-19T03:12:47.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.672+0000 7f640effd700 1 -- 172.21.15.59:0/4240878217 learned_addr learned my addr 172.21.15.59:0/4240878217 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:47.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.672+0000 7f640f7fe700 1 --2- 172.21.15.59:0/4240878217 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64100fde90 0x7f6410115db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:47.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.672+0000 7f640effd700 1 -- 172.21.15.59:0/4240878217 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64100fde90 msgr2=0x7f6410115db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:47.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.672+0000 7f640effd700 1 --2- 172.21.15.59:0/4240878217 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64100fde90 0x7f6410115db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:47.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.672+0000 7f640effd700 1 -- 172.21.15.59:0/4240878217 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f8005040 con 0x7f64100fe830 2024-03-19T03:12:47.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.672+0000 7f640f7fe700 1 --2- 172.21.15.59:0/4240878217 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64100fde90 0x7f6410115db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:12:47.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.672+0000 7f640effd700 1 --2- 172.21.15.59:0/4240878217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64100fe830 0x7f64101162f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f63f800b3b0 tx=0x7f63f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:47.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.673+0000 7f6407fff700 1 -- 172.21.15.59:0/4240878217 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63f800e040 con 0x7f64100fe830 2024-03-19T03:12:47.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.673+0000 7f64161b5700 1 -- 172.21.15.59:0/4240878217 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f641010ff90 con 0x7f64100fe830 2024-03-19T03:12:47.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.673+0000 7f6407fff700 1 -- 172.21.15.59:0/4240878217 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f63f8013420 con 0x7f64100fe830 2024-03-19T03:12:47.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.673+0000 7f6407fff700 1 -- 172.21.15.59:0/4240878217 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63f801c610 con 0x7f64100fe830 2024-03-19T03:12:47.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.673+0000 7f64161b5700 1 -- 172.21.15.59:0/4240878217 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64101104d0 con 0x7f64100fe830 2024-03-19T03:12:47.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.674+0000 7f64161b5700 1 -- 172.21.15.59:0/4240878217 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6410106a20 con 0x7f64100fe830 2024-03-19T03:12:47.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.674+0000 7f6407fff700 1 -- 172.21.15.59:0/4240878217 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f63f8013590 con 0x7f64100fe830 2024-03-19T03:12:47.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.675+0000 7f6407fff700 1 --2- 172.21.15.59:0/4240878217 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f63fc0692a0 0x7f63fc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:47.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.675+0000 7f6407fff700 1 -- 172.21.15.59:0/4240878217 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f63f8088170 con 0x7f64100fe830 2024-03-19T03:12:47.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.675+0000 7f640f7fe700 1 --2- 172.21.15.59:0/4240878217 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f63fc0692a0 0x7f63fc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:47.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.675+0000 7f640f7fe700 1 --2- 172.21.15.59:0/4240878217 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f63fc0692a0 0x7f63fc06b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6400009fd0 tx=0x7f6400009d70 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:47.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.679+0000 7f6407fff700 1 -- 172.21.15.59:0/4240878217 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f63f8058210 con 0x7f64100fe830 2024-03-19T03:12:47.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.874+0000 7f64161b5700 1 -- 172.21.15.59:0/4240878217 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f6410065e30 con 0x7f64100fe830 2024-03-19T03:12:47.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.875+0000 7f6407fff700 1 -- 172.21.15.59:0/4240878217 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f63f8013840 con 0x7f64100fe830 2024-03-19T03:12:47.877 INFO:teuthology.orchestra.run.smithi059.stdout:146028888081 2024-03-19T03:12:47.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.876+0000 7f64161b5700 1 -- 172.21.15.59:0/4240878217 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f63fc0692a0 msgr2=0x7f63fc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:47.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.876+0000 7f64161b5700 1 --2- 172.21.15.59:0/4240878217 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f63fc0692a0 0x7f63fc06b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6400009fd0 tx=0x7f6400009d70 comp rx=0 tx=0).stop 2024-03-19T03:12:47.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.876+0000 7f64161b5700 1 -- 172.21.15.59:0/4240878217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64100fe830 msgr2=0x7f64101162f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:47.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.876+0000 7f64161b5700 1 --2- 172.21.15.59:0/4240878217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64100fe830 0x7f64101162f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f63f800b3b0 tx=0x7f63f8009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:47.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.876+0000 7f64161b5700 1 -- 172.21.15.59:0/4240878217 shutdown_connections 2024-03-19T03:12:47.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.876+0000 7f64161b5700 1 --2- 172.21.15.59:0/4240878217 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f63fc0692a0 0x7f63fc06b750 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:47.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.877+0000 7f64161b5700 1 --2- 172.21.15.59:0/4240878217 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64100fde90 0x7f6410115db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:47.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.877+0000 7f64161b5700 1 --2- 172.21.15.59:0/4240878217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64100fe830 0x7f64101162f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:47.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.877+0000 7f64161b5700 1 -- 172.21.15.59:0/4240878217 >> 172.21.15.59:0/4240878217 conn(0x7f6410072bc0 msgr2=0x7f64100fb730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:47.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.877+0000 7f64161b5700 1 -- 172.21.15.59:0/4240878217 shutdown_connections 2024-03-19T03:12:47.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:47.877+0000 7f64161b5700 1 -- 172.21.15.59:0/4240878217 wait complete. 2024-03-19T03:12:48.116 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705701 got 38654705703 for osd.0 2024-03-19T03:12:48.116 DEBUG:teuthology.parallel:result is None 2024-03-19T03:12:48.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:48 smithi059 ceph-mon[29133]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:48.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:48 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/4240878217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-19T03:12:48.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:48 smithi176 ceph-mon[36450]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:48.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:48 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/4240878217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-19T03:12:48.823 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888079 got 146028888081 for osd.4 2024-03-19T03:12:48.823 DEBUG:teuthology.parallel:result is None 2024-03-19T03:12:48.823 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-19T03:12:48.823 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-19T03:12:49.086 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:50.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:50 smithi059 ceph-mon[29133]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:50.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:50 smithi176 ceph-mon[36450]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:50.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.649+0000 7f4fa30b3700 1 -- 172.21.15.59:0/781511364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f9c102e00 msgr2=0x7f4f9c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:50.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.649+0000 7f4fa30b3700 1 --2- 172.21.15.59:0/781511364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f9c102e00 0x7f4f9c1031d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4f84004940 tx=0x7f4f84009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:50.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.649+0000 7f4fa30b3700 1 -- 172.21.15.59:0/781511364 shutdown_connections 2024-03-19T03:12:50.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.649+0000 7f4fa30b3700 1 --2- 172.21.15.59:0/781511364 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f9c1037a0 0x7f4f9c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:50.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.649+0000 7f4fa30b3700 1 --2- 172.21.15.59:0/781511364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f9c102e00 0x7f4f9c1031d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:50.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.649+0000 7f4fa30b3700 1 -- 172.21.15.59:0/781511364 >> 172.21.15.59:0/781511364 conn(0x7f4f9c0f89b0 msgr2=0x7f4f9c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:50.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.649+0000 7f4fa30b3700 1 -- 172.21.15.59:0/781511364 shutdown_connections 2024-03-19T03:12:50.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.649+0000 7f4fa30b3700 1 -- 172.21.15.59:0/781511364 wait complete. 2024-03-19T03:12:50.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.650+0000 7f4fa30b3700 1 Processor -- start 2024-03-19T03:12:50.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.650+0000 7f4fa30b3700 1 -- start start 2024-03-19T03:12:50.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.650+0000 7f4fa30b3700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f9c102e00 0x7f4f9c113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:50.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.650+0000 7f4fa30b3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f9c1037a0 0x7f4f9c113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:50.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.650+0000 7f4fa30b3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c114520 con 0x7f4f9c1037a0 2024-03-19T03:12:50.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.650+0000 7f4fa30b3700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c10db70 con 0x7f4f9c102e00 2024-03-19T03:12:50.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.650+0000 7f4f9bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f9c1037a0 0x7f4f9c113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:50.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.650+0000 7f4f9bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f9c1037a0 0x7f4f9c113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32940/0 (socket says 172.21.15.59:32940) 2024-03-19T03:12:50.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.650+0000 7f4f9bfff700 1 -- 172.21.15.59:0/4145917981 learned_addr learned my addr 172.21.15.59:0/4145917981 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:50.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.651+0000 7f4fa0e4f700 1 --2- 172.21.15.59:0/4145917981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f9c102e00 0x7f4f9c113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:50.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.651+0000 7f4f9bfff700 1 -- 172.21.15.59:0/4145917981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f9c102e00 msgr2=0x7f4f9c113990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:50.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.651+0000 7f4f9bfff700 1 --2- 172.21.15.59:0/4145917981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f9c102e00 0x7f4f9c113990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:50.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.651+0000 7f4f9bfff700 1 -- 172.21.15.59:0/4145917981 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f84005040 con 0x7f4f9c1037a0 2024-03-19T03:12:50.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.651+0000 7f4f9bfff700 1 --2- 172.21.15.59:0/4145917981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f9c1037a0 0x7f4f9c113ed0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00c800 tx=0x7f4f8c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:50.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.651+0000 7f4f997fa700 1 -- 172.21.15.59:0/4145917981 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f8c0106f0 con 0x7f4f9c1037a0 2024-03-19T03:12:50.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.651+0000 7f4f997fa700 1 -- 172.21.15.59:0/4145917981 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f8c010d20 con 0x7f4f9c1037a0 2024-03-19T03:12:50.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.651+0000 7f4fa30b3700 1 -- 172.21.15.59:0/4145917981 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f9c10de50 con 0x7f4f9c1037a0 2024-03-19T03:12:50.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.651+0000 7f4f997fa700 1 -- 172.21.15.59:0/4145917981 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f8c0113a0 con 0x7f4f9c1037a0 2024-03-19T03:12:50.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.651+0000 7f4fa30b3700 1 -- 172.21.15.59:0/4145917981 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f9c10e410 con 0x7f4f9c1037a0 2024-03-19T03:12:50.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.652+0000 7f4fa30b3700 1 -- 172.21.15.59:0/4145917981 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f9c100110 con 0x7f4f9c1037a0 2024-03-19T03:12:50.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.652+0000 7f4f997fa700 1 -- 172.21.15.59:0/4145917981 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4f8c010850 con 0x7f4f9c1037a0 2024-03-19T03:12:50.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.653+0000 7f4f997fa700 1 --2- 172.21.15.59:0/4145917981 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f4f880692a0 0x7f4f8806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:50.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.653+0000 7f4f997fa700 1 -- 172.21.15.59:0/4145917981 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4f8c086e10 con 0x7f4f9c1037a0 2024-03-19T03:12:50.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.653+0000 7f4fa0e4f700 1 --2- 172.21.15.59:0/4145917981 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f4f880692a0 0x7f4f8806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:50.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.653+0000 7f4fa0e4f700 1 --2- 172.21.15.59:0/4145917981 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f4f880692a0 0x7f4f8806b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4f84004f90 tx=0x7f4f84009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:50.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.657+0000 7f4f997fa700 1 -- 172.21.15.59:0/4145917981 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f4f8c056f10 con 0x7f4f9c1037a0 2024-03-19T03:12:50.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.843+0000 7f4fa30b3700 1 -- 172.21.15.59:0/4145917981 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4f9c10f050 con 0x7f4f880692a0 2024-03-19T03:12:50.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.844+0000 7f4f997fa700 1 -- 172.21.15.59:0/4145917981 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28124 (secure 0 0 0) 0x7f4f9c10f050 con 0x7f4f880692a0 2024-03-19T03:12:50.846 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:12:50.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.846+0000 7f4fa30b3700 1 -- 172.21.15.59:0/4145917981 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f4f880692a0 msgr2=0x7f4f8806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:50.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.846+0000 7f4fa30b3700 1 --2- 172.21.15.59:0/4145917981 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f4f880692a0 0x7f4f8806b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4f84004f90 tx=0x7f4f84009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:50.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.846+0000 7f4fa30b3700 1 -- 172.21.15.59:0/4145917981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f9c1037a0 msgr2=0x7f4f9c113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:50.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.846+0000 7f4fa30b3700 1 --2- 172.21.15.59:0/4145917981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f9c1037a0 0x7f4f9c113ed0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00c800 tx=0x7f4f8c004470 comp rx=0 tx=0).stop 2024-03-19T03:12:50.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.846+0000 7f4fa30b3700 1 -- 172.21.15.59:0/4145917981 shutdown_connections 2024-03-19T03:12:50.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.846+0000 7f4fa30b3700 1 --2- 172.21.15.59:0/4145917981 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f4f880692a0 0x7f4f8806b750 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:50.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.846+0000 7f4fa30b3700 1 --2- 172.21.15.59:0/4145917981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f9c102e00 0x7f4f9c113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:50.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.846+0000 7f4fa30b3700 1 --2- 172.21.15.59:0/4145917981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f9c1037a0 0x7f4f9c113ed0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:50.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.846+0000 7f4fa30b3700 1 -- 172.21.15.59:0/4145917981 >> 172.21.15.59:0/4145917981 conn(0x7f4f9c0f89b0 msgr2=0x7f4f9c1073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:50.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.846+0000 7f4fa30b3700 1 -- 172.21.15.59:0/4145917981 shutdown_connections 2024-03-19T03:12:50.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:50.846+0000 7f4fa30b3700 1 -- 172.21.15.59:0/4145917981 wait complete. 2024-03-19T03:12:50.849 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-19T03:12:51.553 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:51 smithi059 ceph-mon[29133]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T03:12:51.553 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:51 smithi059 ceph-mon[29133]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:51.581 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-03-19T03:12:48.954937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787604,"kb_used_data":3876,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683308,"statfs":{"total":575970213888,"available":574139707392,"internally_reserved":0,"allocated":3969024,"data_stored":2603946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006288"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":189,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-19T03:11:58.265892+0000","last_change":"2024-03-19T03:11:33.221487+0000","last_active":"2024-03-19T03:11:58.265892+0000","last_peered":"2024-03-19T03:11:58.265892+0000","last_clean":"2024-03-19T03:11:58.265892+0000","last_became_active":"2024-03-19T03:11:33.221211+0000","last_became_peered":"2024-03-19T03:11:33.221211+0000","last_unstale":"2024-03-19T03:11:58.265892+0000","last_undegraded":"2024-03-19T03:11:58.265892+0000","last_fullsized":"2024-03-19T03:11:58.265892+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:10:45.747873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:10:45.747873+0000","last_clean_scrub_stamp":"2024-03-19T03:10:45.747873+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:34:25.086969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":138807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":138807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 19 03:12:35 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.56100000000000005}]},{"osd":1,"last update":"Tue Mar 19 03:12:35 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.57799999999999996}]},{"osd":2,"last update":"Tue Mar 19 03:12:35 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.497},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":16.515999999999998,"5min":16.515999999999998,"15min":16.515999999999998},"last":0.52600000000000002}]},{"osd":3,"last update":"Tue Mar 19 03:12:35 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.38400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 19 03:12:08 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.47799999999999998}]},{"osd":1,"last update":"Tue Mar 19 03:12:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.51800000000000002}]},{"osd":2,"last update":"Tue Mar 19 03:12:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.5},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.56599999999999995}]},{"osd":4,"last update":"Tue Mar 19 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.46100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":138807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 19 03:11:47 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.41899999999999998}]},{"osd":1,"last update":"Tue Mar 19 03:11:47 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.436},{"interface":"front","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.45300000000000001}]},{"osd":3,"last update":"Tue Mar 19 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.626}]},{"osd":4,"last update":"Tue Mar 19 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.57699999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Mar 19 03:12:19 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.29499999999999998,"15min":0.28599999999999998},"min":{"1min":0.22800000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.498,"5min":0.498,"15min":0.498},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.308,"15min":0.29999999999999999},"min":{"1min":0.249,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.39200000000000002}]},{"osd":2,"last update":"Tue Mar 19 03:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.46999999999999997}]},{"osd":3,"last update":"Tue Mar 19 03:12:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.60399999999999998}]},{"osd":4,"last update":"Tue Mar 19 03:12:36 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.64800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 19 03:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.33300000000000002,"15min":0.32900000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.33800000000000002,"15min":0.33600000000000002},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.44700000000000001,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.379}]},{"osd":2,"last update":"Tue Mar 19 03:11:45 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"last":0.36099999999999999}]},{"osd":3,"last update":"Tue Mar 19 03:12:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.47799999999999998}]},{"osd":4,"last update":"Tue Mar 19 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":8.1620000000000008,"5min":8.1620000000000008,"15min":8.1620000000000008},"last":0.39900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T03:12:51.583 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-19T03:12:51.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:51 smithi176 ceph-mon[36450]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T03:12:51.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:51 smithi176 ceph-mon[36450]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:51.847 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:53.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.428+0000 7fe587442700 1 -- 172.21.15.59:0/3098815117 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe580102680 msgr2=0x7fe5800fbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:53.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.428+0000 7fe587442700 1 --2- 172.21.15.59:0/3098815117 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe580102680 0x7fe5800fbcf0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe574004940 tx=0x7fe574009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:53.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.428+0000 7fe587442700 1 -- 172.21.15.59:0/3098815117 shutdown_connections 2024-03-19T03:12:53.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.428+0000 7fe587442700 1 --2- 172.21.15.59:0/3098815117 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe580102680 0x7fe5800fbcf0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:53.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.428+0000 7fe587442700 1 --2- 172.21.15.59:0/3098815117 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe580101ce0 0x7fe5801020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:53.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.428+0000 7fe587442700 1 -- 172.21.15.59:0/3098815117 >> 172.21.15.59:0/3098815117 conn(0x7fe5800f7890 msgr2=0x7fe5800f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:53.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.428+0000 7fe587442700 1 -- 172.21.15.59:0/3098815117 shutdown_connections 2024-03-19T03:12:53.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.428+0000 7fe587442700 1 -- 172.21.15.59:0/3098815117 wait complete. 2024-03-19T03:12:53.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.429+0000 7fe587442700 1 Processor -- start 2024-03-19T03:12:53.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.429+0000 7fe587442700 1 -- start start 2024-03-19T03:12:53.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.429+0000 7fe587442700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe580101ce0 0x7fe58010eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:53.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.429+0000 7fe587442700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe580102680 0x7fe58010f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:53.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.429+0000 7fe587442700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58010f5d0 con 0x7fe580101ce0 2024-03-19T03:12:53.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.429+0000 7fe587442700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58010f710 con 0x7fe580102680 2024-03-19T03:12:53.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.429+0000 7fe5851de700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe580101ce0 0x7fe58010eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:53.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.429+0000 7fe5851de700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe580101ce0 0x7fe58010eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32950/0 (socket says 172.21.15.59:32950) 2024-03-19T03:12:53.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.429+0000 7fe5851de700 1 -- 172.21.15.59:0/3466870397 learned_addr learned my addr 172.21.15.59:0/3466870397 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:53.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.429+0000 7fe5849dd700 1 --2- 172.21.15.59:0/3466870397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe580102680 0x7fe58010f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:53.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.429+0000 7fe5851de700 1 -- 172.21.15.59:0/3466870397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe580102680 msgr2=0x7fe58010f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:53.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.429+0000 7fe5851de700 1 --2- 172.21.15.59:0/3466870397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe580102680 0x7fe58010f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:53.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.429+0000 7fe5851de700 1 -- 172.21.15.59:0/3466870397 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe574005040 con 0x7fe580101ce0 2024-03-19T03:12:53.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.430+0000 7fe5851de700 1 --2- 172.21.15.59:0/3466870397 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe580101ce0 0x7fe58010eb50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe57800bbb0 tx=0x7fe578009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:53.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.430+0000 7fe57dffb700 1 -- 172.21.15.59:0/3466870397 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5780092e0 con 0x7fe580101ce0 2024-03-19T03:12:53.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.430+0000 7fe587442700 1 -- 172.21.15.59:0/3466870397 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe580117bc0 con 0x7fe580101ce0 2024-03-19T03:12:53.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.430+0000 7fe587442700 1 -- 172.21.15.59:0/3466870397 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe580117f90 con 0x7fe580101ce0 2024-03-19T03:12:53.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.430+0000 7fe57dffb700 1 -- 172.21.15.59:0/3466870397 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe57800cb20 con 0x7fe580101ce0 2024-03-19T03:12:53.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.430+0000 7fe57dffb700 1 -- 172.21.15.59:0/3466870397 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe578015d70 con 0x7fe580101ce0 2024-03-19T03:12:53.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.431+0000 7fe587442700 1 -- 172.21.15.59:0/3466870397 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe56c0052b0 con 0x7fe580101ce0 2024-03-19T03:12:53.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.432+0000 7fe57dffb700 1 -- 172.21.15.59:0/3466870397 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe578012070 con 0x7fe580101ce0 2024-03-19T03:12:53.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.432+0000 7fe57dffb700 1 --2- 172.21.15.59:0/3466870397 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe5680692a0 0x7fe56806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:53.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.432+0000 7fe57dffb700 1 -- 172.21.15.59:0/3466870397 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe5780868c0 con 0x7fe580101ce0 2024-03-19T03:12:53.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.432+0000 7fe5849dd700 1 --2- 172.21.15.59:0/3466870397 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe5680692a0 0x7fe56806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:53.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.432+0000 7fe5849dd700 1 --2- 172.21.15.59:0/3466870397 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe5680692a0 0x7fe56806b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe574004890 tx=0x7fe574009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:53.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.436+0000 7fe57dffb700 1 -- 172.21.15.59:0/3466870397 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fe578052f30 con 0x7fe580101ce0 2024-03-19T03:12:53.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.622+0000 7fe587442700 1 -- 172.21.15.59:0/3466870397 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe56c000bc0 con 0x7fe5680692a0 2024-03-19T03:12:53.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.624+0000 7fe57dffb700 1 -- 172.21.15.59:0/3466870397 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28167 (secure 0 0 0) 0x7fe56c000bc0 con 0x7fe5680692a0 2024-03-19T03:12:53.626 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:12:53.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.626+0000 7fe587442700 1 -- 172.21.15.59:0/3466870397 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe5680692a0 msgr2=0x7fe56806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:53.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.626+0000 7fe587442700 1 --2- 172.21.15.59:0/3466870397 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe5680692a0 0x7fe56806b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe574004890 tx=0x7fe574009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:53.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.626+0000 7fe587442700 1 -- 172.21.15.59:0/3466870397 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe580101ce0 msgr2=0x7fe58010eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:53.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.626+0000 7fe587442700 1 --2- 172.21.15.59:0/3466870397 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe580101ce0 0x7fe58010eb50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe57800bbb0 tx=0x7fe578009840 comp rx=0 tx=0).stop 2024-03-19T03:12:53.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.626+0000 7fe587442700 1 -- 172.21.15.59:0/3466870397 shutdown_connections 2024-03-19T03:12:53.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.626+0000 7fe587442700 1 --2- 172.21.15.59:0/3466870397 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe5680692a0 0x7fe56806b750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:53.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.626+0000 7fe587442700 1 --2- 172.21.15.59:0/3466870397 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe580101ce0 0x7fe58010eb50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:53.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.626+0000 7fe587442700 1 --2- 172.21.15.59:0/3466870397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe580102680 0x7fe58010f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:53.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.626+0000 7fe587442700 1 -- 172.21.15.59:0/3466870397 >> 172.21.15.59:0/3466870397 conn(0x7fe5800f7890 msgr2=0x7fe580104a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:53.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.626+0000 7fe587442700 1 -- 172.21.15.59:0/3466870397 shutdown_connections 2024-03-19T03:12:53.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:53.626+0000 7fe587442700 1 -- 172.21.15.59:0/3466870397 wait complete. 2024-03-19T03:12:53.629 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-19T03:12:54.182 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:54 smithi059 ceph-mon[29133]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:54.207 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-03-19T03:12:52.955474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787604,"kb_used_data":3876,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683308,"statfs":{"total":575970213888,"available":574139707392,"internally_reserved":0,"allocated":3969024,"data_stored":2603946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006190"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":189,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-19T03:11:58.265892+0000","last_change":"2024-03-19T03:11:33.221487+0000","last_active":"2024-03-19T03:11:58.265892+0000","last_peered":"2024-03-19T03:11:58.265892+0000","last_clean":"2024-03-19T03:11:58.265892+0000","last_became_active":"2024-03-19T03:11:33.221211+0000","last_became_peered":"2024-03-19T03:11:33.221211+0000","last_unstale":"2024-03-19T03:11:58.265892+0000","last_undegraded":"2024-03-19T03:11:58.265892+0000","last_fullsized":"2024-03-19T03:11:58.265892+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:10:45.747873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:10:45.747873+0000","last_clean_scrub_stamp":"2024-03-19T03:10:45.747873+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:34:25.086969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":138807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":138807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 19 03:12:35 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.56100000000000005}]},{"osd":1,"last update":"Tue Mar 19 03:12:35 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.57799999999999996}]},{"osd":2,"last update":"Tue Mar 19 03:12:35 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.497},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":16.515999999999998,"5min":16.515999999999998,"15min":16.515999999999998},"last":0.52600000000000002}]},{"osd":3,"last update":"Tue Mar 19 03:12:35 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.38400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 19 03:12:08 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.51900000000000002}]},{"osd":1,"last update":"Tue Mar 19 03:12:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.498}]},{"osd":2,"last update":"Tue Mar 19 03:12:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.53200000000000003}]},{"osd":4,"last update":"Tue Mar 19 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.37}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":138807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 19 03:11:47 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.32400000000000001}]},{"osd":1,"last update":"Tue Mar 19 03:11:47 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.308}]},{"osd":3,"last update":"Tue Mar 19 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.48999999999999999}]},{"osd":4,"last update":"Tue Mar 19 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.56699999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705705,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Mar 19 03:12:19 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.29499999999999998,"15min":0.28599999999999998},"min":{"1min":0.22800000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.498,"5min":0.498,"15min":0.498},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.308,"15min":0.29999999999999999},"min":{"1min":0.249,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.26500000000000001}]},{"osd":2,"last update":"Tue Mar 19 03:12:50 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.35099999999999998,"15min":0.35699999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.45100000000000001,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.251},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.36699999999999999,"15min":0.373},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.46999999999999997,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.30499999999999999}]},{"osd":3,"last update":"Tue Mar 19 03:12:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.438}]},{"osd":4,"last update":"Tue Mar 19 03:12:36 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.40899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 19 03:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.33300000000000002,"15min":0.32900000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.33800000000000002,"15min":0.33600000000000002},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.44700000000000001,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.379}]},{"osd":2,"last update":"Tue Mar 19 03:11:45 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"last":0.36099999999999999}]},{"osd":3,"last update":"Tue Mar 19 03:12:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.47799999999999998}]},{"osd":4,"last update":"Tue Mar 19 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":8.1620000000000008,"5min":8.1620000000000008,"15min":8.1620000000000008},"last":0.39900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T03:12:54.208 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-19T03:12:54.208 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-19T03:12:54.208 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-19T03:12:54.208 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-19T03:12:54.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:54 smithi176 ceph-mon[36450]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:54.468 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:55.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:55 smithi059 ceph-mon[29133]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T03:12:55.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:55 smithi176 ceph-mon[36450]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T03:12:56.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.108+0000 7f51a298a700 1 -- 172.21.15.59:0/3165471779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f519c0fde50 msgr2=0x7f519c0fe220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:56.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.108+0000 7f51a298a700 1 --2- 172.21.15.59:0/3165471779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f519c0fde50 0x7f519c0fe220 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f518c004970 tx=0x7f518c009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:56.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.108+0000 7f51a298a700 1 -- 172.21.15.59:0/3165471779 shutdown_connections 2024-03-19T03:12:56.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.108+0000 7f51a298a700 1 --2- 172.21.15.59:0/3165471779 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f519c0fe7f0 0x7f519c106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:56.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.108+0000 7f51a298a700 1 --2- 172.21.15.59:0/3165471779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f519c0fde50 0x7f519c0fe220 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:56.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.108+0000 7f51a298a700 1 -- 172.21.15.59:0/3165471779 >> 172.21.15.59:0/3165471779 conn(0x7f519c0f7880 msgr2=0x7f519c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:56.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.108+0000 7f51a298a700 1 -- 172.21.15.59:0/3165471779 shutdown_connections 2024-03-19T03:12:56.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.108+0000 7f51a298a700 1 -- 172.21.15.59:0/3165471779 wait complete. 2024-03-19T03:12:56.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.108+0000 7f51a298a700 1 Processor -- start 2024-03-19T03:12:56.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.109+0000 7f51a298a700 1 -- start start 2024-03-19T03:12:56.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.109+0000 7f51a298a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f519c0fde50 0x7f519c113e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:56.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.109+0000 7f51a298a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f519c0fe7f0 0x7f519c1143b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:56.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.109+0000 7f51a298a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519c114a00 con 0x7f519c0fe7f0 2024-03-19T03:12:56.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.109+0000 7f51a298a700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519c10ddd0 con 0x7f519c0fde50 2024-03-19T03:12:56.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.109+0000 7f519b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f519c0fe7f0 0x7f519c1143b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:56.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.109+0000 7f519b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f519c0fe7f0 0x7f519c1143b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48530/0 (socket says 172.21.15.59:48530) 2024-03-19T03:12:56.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.109+0000 7f519b7fe700 1 -- 172.21.15.59:0/2060230048 learned_addr learned my addr 172.21.15.59:0/2060230048 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:56.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.109+0000 7f519bfff700 1 --2- 172.21.15.59:0/2060230048 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f519c0fde50 0x7f519c113e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:56.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.109+0000 7f519b7fe700 1 -- 172.21.15.59:0/2060230048 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f519c0fde50 msgr2=0x7f519c113e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:56.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.109+0000 7f519b7fe700 1 --2- 172.21.15.59:0/2060230048 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f519c0fde50 0x7f519c113e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:56.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.109+0000 7f519b7fe700 1 -- 172.21.15.59:0/2060230048 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518c005040 con 0x7f519c0fe7f0 2024-03-19T03:12:56.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.109+0000 7f519bfff700 1 --2- 172.21.15.59:0/2060230048 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f519c0fde50 0x7f519c113e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:12:56.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.109+0000 7f519b7fe700 1 --2- 172.21.15.59:0/2060230048 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f519c0fe7f0 0x7f519c1143b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f519000c8c0 tx=0x7f51900044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:56.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.109+0000 7f5198ff9700 1 -- 172.21.15.59:0/2060230048 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f519000a980 con 0x7f519c0fe7f0 2024-03-19T03:12:56.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.110+0000 7f5198ff9700 1 -- 172.21.15.59:0/2060230048 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f519000ce60 con 0x7f519c0fe7f0 2024-03-19T03:12:56.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.110+0000 7f51a298a700 1 -- 172.21.15.59:0/2060230048 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f519c10e0b0 con 0x7f519c0fe7f0 2024-03-19T03:12:56.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.110+0000 7f5198ff9700 1 -- 172.21.15.59:0/2060230048 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51900115e0 con 0x7f519c0fe7f0 2024-03-19T03:12:56.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.110+0000 7f51a298a700 1 -- 172.21.15.59:0/2060230048 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f519c10e5f0 con 0x7f519c0fe7f0 2024-03-19T03:12:56.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.110+0000 7f51a298a700 1 -- 172.21.15.59:0/2060230048 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f519c1017c0 con 0x7f519c0fe7f0 2024-03-19T03:12:56.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.111+0000 7f5198ff9700 1 -- 172.21.15.59:0/2060230048 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f5190012400 con 0x7f519c0fe7f0 2024-03-19T03:12:56.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.111+0000 7f5198ff9700 1 --2- 172.21.15.59:0/2060230048 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f51840692a0 0x7f518406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:56.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.111+0000 7f5198ff9700 1 -- 172.21.15.59:0/2060230048 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5190086e50 con 0x7f519c0fe7f0 2024-03-19T03:12:56.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.111+0000 7f519bfff700 1 --2- 172.21.15.59:0/2060230048 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f51840692a0 0x7f518406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:56.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.112+0000 7f519bfff700 1 --2- 172.21.15.59:0/2060230048 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f51840692a0 0x7f518406b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f518c004f50 tx=0x7f518c009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:56.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.116+0000 7f5198ff9700 1 -- 172.21.15.59:0/2060230048 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f5190056ff0 con 0x7f519c0fe7f0 2024-03-19T03:12:56.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:56 smithi059 ceph-mon[29133]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:56.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.352+0000 7f51a298a700 1 -- 172.21.15.59:0/2060230048 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f519c066730 con 0x7f519c0fe7f0 2024-03-19T03:12:56.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.352+0000 7f5198ff9700 1 -- 172.21.15.59:0/2060230048 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f51900126b0 con 0x7f519c0fe7f0 2024-03-19T03:12:56.354 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:12:56.355 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-19T03:12:56.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.354+0000 7f51a298a700 1 -- 172.21.15.59:0/2060230048 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f51840692a0 msgr2=0x7f518406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:56.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.354+0000 7f51a298a700 1 --2- 172.21.15.59:0/2060230048 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f51840692a0 0x7f518406b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f518c004f50 tx=0x7f518c009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:56.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.354+0000 7f51a298a700 1 -- 172.21.15.59:0/2060230048 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f519c0fe7f0 msgr2=0x7f519c1143b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:56.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.354+0000 7f51a298a700 1 --2- 172.21.15.59:0/2060230048 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f519c0fe7f0 0x7f519c1143b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f519000c8c0 tx=0x7f51900044c0 comp rx=0 tx=0).stop 2024-03-19T03:12:56.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.354+0000 7f51a298a700 1 -- 172.21.15.59:0/2060230048 shutdown_connections 2024-03-19T03:12:56.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.354+0000 7f51a298a700 1 --2- 172.21.15.59:0/2060230048 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f51840692a0 0x7f518406b750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:56.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.354+0000 7f51a298a700 1 --2- 172.21.15.59:0/2060230048 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f519c0fde50 0x7f519c113e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:56.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.354+0000 7f51a298a700 1 --2- 172.21.15.59:0/2060230048 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f519c0fe7f0 0x7f519c1143b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:56.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.354+0000 7f51a298a700 1 -- 172.21.15.59:0/2060230048 >> 172.21.15.59:0/2060230048 conn(0x7f519c0f7880 msgr2=0x7f519c0ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:56.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.354+0000 7f51a298a700 1 -- 172.21.15.59:0/2060230048 shutdown_connections 2024-03-19T03:12:56.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:56.354+0000 7f51a298a700 1 -- 172.21.15.59:0/2060230048 wait complete. 2024-03-19T03:12:56.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:56 smithi176 ceph-mon[36450]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:56.945 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-19T03:12:56.946 INFO:tasks.cephadm:Setup complete, yielding 2024-03-19T03:12:56.946 INFO:teuthology.run_tasks:Running task print... 2024-03-19T03:12:56.955 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-03-19T03:12:56.955 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-19T03:12:56.999 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2024-03-19T03:12:56.999 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-03-19T03:12:57.261 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:12:57.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:57 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/2060230048' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-19T03:12:57.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:57 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/2060230048' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-19T03:12:58.318 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:12:58 smithi059 ceph-mon[29133]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:58.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:12:58 smithi176 ceph-mon[36450]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:12:58.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.836+0000 7fa47f80f700 1 -- 172.21.15.59:0/1604899191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4780fcef0 msgr2=0x7fa4780fd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:58.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.836+0000 7fa47f80f700 1 --2- 172.21.15.59:0/1604899191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4780fcef0 0x7fa4780fd300 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa474004940 tx=0x7fa474009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:58.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.836+0000 7fa47f80f700 1 -- 172.21.15.59:0/1604899191 shutdown_connections 2024-03-19T03:12:58.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.836+0000 7fa47f80f700 1 --2- 172.21.15.59:0/1604899191 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4780fe060 0x7fa4780fe4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:58.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.836+0000 7fa47f80f700 1 --2- 172.21.15.59:0/1604899191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4780fcef0 0x7fa4780fd300 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:58.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.836+0000 7fa47f80f700 1 -- 172.21.15.59:0/1604899191 >> 172.21.15.59:0/1604899191 conn(0x7fa4780f85b0 msgr2=0x7fa4780fa9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:58.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.836+0000 7fa47f80f700 1 -- 172.21.15.59:0/1604899191 shutdown_connections 2024-03-19T03:12:58.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.836+0000 7fa47f80f700 1 -- 172.21.15.59:0/1604899191 wait complete. 2024-03-19T03:12:58.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.837+0000 7fa47f80f700 1 Processor -- start 2024-03-19T03:12:58.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.837+0000 7fa47f80f700 1 -- start start 2024-03-19T03:12:58.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.837+0000 7fa47f80f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4780fcef0 0x7fa47810da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:58.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.837+0000 7fa47f80f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4780fe060 0x7fa47810dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:58.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.837+0000 7fa47f80f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47810e630 con 0x7fa4780fe060 2024-03-19T03:12:58.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.837+0000 7fa47f80f700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47810ee20 con 0x7fa4780fcef0 2024-03-19T03:12:58.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.837+0000 7fa47cdaa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4780fe060 0x7fa47810dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:58.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.837+0000 7fa47cdaa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4780fe060 0x7fa47810dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48548/0 (socket says 172.21.15.59:48548) 2024-03-19T03:12:58.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.837+0000 7fa47cdaa700 1 -- 172.21.15.59:0/3281519430 learned_addr learned my addr 172.21.15.59:0/3281519430 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:12:58.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.837+0000 7fa47d5ab700 1 --2- 172.21.15.59:0/3281519430 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4780fcef0 0x7fa47810da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:58.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.837+0000 7fa47cdaa700 1 -- 172.21.15.59:0/3281519430 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4780fcef0 msgr2=0x7fa47810da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:58.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.837+0000 7fa47cdaa700 1 --2- 172.21.15.59:0/3281519430 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4780fcef0 0x7fa47810da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:58.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.837+0000 7fa47cdaa700 1 -- 172.21.15.59:0/3281519430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa474005040 con 0x7fa4780fe060 2024-03-19T03:12:58.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.837+0000 7fa47d5ab700 1 --2- 172.21.15.59:0/3281519430 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4780fcef0 0x7fa47810da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:12:58.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.837+0000 7fa47cdaa700 1 --2- 172.21.15.59:0/3281519430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4780fe060 0x7fa47810dfa0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa46800b790 tx=0x7fa468009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:58.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.838+0000 7fa46dffb700 1 -- 172.21.15.59:0/3281519430 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa46800f770 con 0x7fa4780fe060 2024-03-19T03:12:58.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.838+0000 7fa46dffb700 1 -- 172.21.15.59:0/3281519430 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa46800fda0 con 0x7fa4780fe060 2024-03-19T03:12:58.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.838+0000 7fa47f80f700 1 -- 172.21.15.59:0/3281519430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa47810f100 con 0x7fa4780fe060 2024-03-19T03:12:58.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.838+0000 7fa46dffb700 1 -- 172.21.15.59:0/3281519430 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa46800d310 con 0x7fa4780fe060 2024-03-19T03:12:58.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.838+0000 7fa47f80f700 1 -- 172.21.15.59:0/3281519430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa47810f6c0 con 0x7fa4780fe060 2024-03-19T03:12:58.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.839+0000 7fa47f80f700 1 -- 172.21.15.59:0/3281519430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa47805f5f0 con 0x7fa4780fe060 2024-03-19T03:12:58.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.839+0000 7fa46dffb700 1 -- 172.21.15.59:0/3281519430 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa468010420 con 0x7fa4780fe060 2024-03-19T03:12:58.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.839+0000 7fa46dffb700 1 --2- 172.21.15.59:0/3281519430 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fa4640692a0 0x7fa46406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:12:58.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.840+0000 7fa46dffb700 1 -- 172.21.15.59:0/3281519430 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa468086da0 con 0x7fa4780fe060 2024-03-19T03:12:58.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.840+0000 7fa47d5ab700 1 --2- 172.21.15.59:0/3281519430 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fa4640692a0 0x7fa46406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:12:58.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.840+0000 7fa47d5ab700 1 --2- 172.21.15.59:0/3281519430 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fa4640692a0 0x7fa46406b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa47400b3b0 tx=0x7fa474009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:12:58.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:58.844+0000 7fa46dffb700 1 -- 172.21.15.59:0/3281519430 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fa468056f40 con 0x7fa4780fe060 2024-03-19T03:12:59.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:59.036+0000 7fa47f80f700 1 -- 172.21.15.59:0/3281519430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7fa47804bed0 con 0x7fa4780fe060 2024-03-19T03:12:59.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:59.042+0000 7fa46dffb700 1 -- 172.21.15.59:0/3281519430 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7fa47804bed0 con 0x7fa4780fe060 2024-03-19T03:12:59.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:59.044+0000 7fa47f80f700 1 -- 172.21.15.59:0/3281519430 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fa4640692a0 msgr2=0x7fa46406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:59.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:59.044+0000 7fa47f80f700 1 --2- 172.21.15.59:0/3281519430 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fa4640692a0 0x7fa46406b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa47400b3b0 tx=0x7fa474009e30 comp rx=0 tx=0).stop 2024-03-19T03:12:59.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:59.044+0000 7fa47f80f700 1 -- 172.21.15.59:0/3281519430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4780fe060 msgr2=0x7fa47810dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:12:59.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:59.044+0000 7fa47f80f700 1 --2- 172.21.15.59:0/3281519430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4780fe060 0x7fa47810dfa0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa46800b790 tx=0x7fa468009250 comp rx=0 tx=0).stop 2024-03-19T03:12:59.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:59.044+0000 7fa47f80f700 1 -- 172.21.15.59:0/3281519430 shutdown_connections 2024-03-19T03:12:59.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:59.044+0000 7fa47f80f700 1 --2- 172.21.15.59:0/3281519430 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fa4640692a0 0x7fa46406b750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:59.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:59.044+0000 7fa47f80f700 1 --2- 172.21.15.59:0/3281519430 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4780fcef0 0x7fa47810da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:59.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:59.044+0000 7fa47f80f700 1 --2- 172.21.15.59:0/3281519430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4780fe060 0x7fa47810dfa0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:12:59.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:59.044+0000 7fa47f80f700 1 -- 172.21.15.59:0/3281519430 >> 172.21.15.59:0/3281519430 conn(0x7fa4780f85b0 msgr2=0x7fa4780f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:12:59.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:59.044+0000 7fa47f80f700 1 -- 172.21.15.59:0/3281519430 shutdown_connections 2024-03-19T03:12:59.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:12:59.044+0000 7fa47f80f700 1 -- 172.21.15.59:0/3281519430 wait complete. 2024-03-19T03:12:59.616 INFO:teuthology.run_tasks:Running task print... 2024-03-19T03:12:59.626 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-03-19T03:12:59.627 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-19T03:12:59.664 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2024-03-19T03:12:59.665 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-19T03:12:59.939 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:13:00.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:00 smithi059 ceph-mon[29133]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:00.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:00 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/3281519430' entity='client.admin' 2024-03-19T03:13:00.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:00 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:13:00.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:00 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:13:00.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:00 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:13:00.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:00 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:00.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:00 smithi176 ceph-mon[36450]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:00.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:00 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/3281519430' entity='client.admin' 2024-03-19T03:13:00.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:00 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:13:00.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:00 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:13:00.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:00 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:13:00.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:00 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:01.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.434+0000 7fef022a3700 1 -- 172.21.15.59:0/3820398828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feefc0fcf80 msgr2=0x7feefc0fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:01.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.434+0000 7fef022a3700 1 --2- 172.21.15.59:0/3820398828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feefc0fcf80 0x7feefc0fd3f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7feee4004940 tx=0x7feee4009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:01.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.434+0000 7fef022a3700 1 -- 172.21.15.59:0/3820398828 shutdown_connections 2024-03-19T03:13:01.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.434+0000 7fef022a3700 1 --2- 172.21.15.59:0/3820398828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feefc0fcf80 0x7feefc0fd3f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:01.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.434+0000 7fef022a3700 1 --2- 172.21.15.59:0/3820398828 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feefc0fbcd0 0x7feefc0fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:01.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.434+0000 7fef022a3700 1 -- 172.21.15.59:0/3820398828 >> 172.21.15.59:0/3820398828 conn(0x7feefc0f7430 msgr2=0x7feefc0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:01.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.434+0000 7fef022a3700 1 -- 172.21.15.59:0/3820398828 shutdown_connections 2024-03-19T03:13:01.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.434+0000 7fef022a3700 1 -- 172.21.15.59:0/3820398828 wait complete. 2024-03-19T03:13:01.437 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:01 smithi059 ceph-mon[29133]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:01.437 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:13:01.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.435+0000 7fef022a3700 1 Processor -- start 2024-03-19T03:13:01.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.435+0000 7fef022a3700 1 -- start start 2024-03-19T03:13:01.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.435+0000 7fef022a3700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feefc0fbcd0 0x7feefc10da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:01.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.435+0000 7fef022a3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feefc0fcf80 0x7feefc10df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:01.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.435+0000 7fef022a3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feefc10e550 con 0x7feefc0fcf80 2024-03-19T03:13:01.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.435+0000 7fef022a3700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feefc10eda0 con 0x7feefc0fbcd0 2024-03-19T03:13:01.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.435+0000 7feefb7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feefc0fcf80 0x7feefc10df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:01.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.435+0000 7feefb7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feefc0fcf80 0x7feefc10df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48570/0 (socket says 172.21.15.59:48570) 2024-03-19T03:13:01.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.435+0000 7feefb7fe700 1 -- 172.21.15.59:0/2217939978 learned_addr learned my addr 172.21.15.59:0/2217939978 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:13:01.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.435+0000 7feefbfff700 1 --2- 172.21.15.59:0/2217939978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feefc0fbcd0 0x7feefc10da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:01.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.435+0000 7feefb7fe700 1 -- 172.21.15.59:0/2217939978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feefc0fbcd0 msgr2=0x7feefc10da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:01.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.435+0000 7feefb7fe700 1 --2- 172.21.15.59:0/2217939978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feefc0fbcd0 0x7feefc10da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:01.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.435+0000 7feefb7fe700 1 -- 172.21.15.59:0/2217939978 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee4005040 con 0x7feefc0fcf80 2024-03-19T03:13:01.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.436+0000 7feefb7fe700 1 --2- 172.21.15.59:0/2217939978 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feefc0fcf80 0x7feefc10df50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7feee400b3b0 tx=0x7feee4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:01.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.436+0000 7feef8ff9700 1 -- 172.21.15.59:0/2217939978 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feee400e040 con 0x7feefc0fcf80 2024-03-19T03:13:01.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.436+0000 7fef022a3700 1 -- 172.21.15.59:0/2217939978 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feefc10f020 con 0x7feefc0fcf80 2024-03-19T03:13:01.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.436+0000 7feef8ff9700 1 -- 172.21.15.59:0/2217939978 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feee400bbf0 con 0x7feefc0fcf80 2024-03-19T03:13:01.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.436+0000 7fef022a3700 1 -- 172.21.15.59:0/2217939978 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feefc10f560 con 0x7feefc0fcf80 2024-03-19T03:13:01.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.436+0000 7feef8ff9700 1 -- 172.21.15.59:0/2217939978 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feee401c630 con 0x7feefc0fcf80 2024-03-19T03:13:01.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.437+0000 7fef022a3700 1 -- 172.21.15.59:0/2217939978 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feefc101b10 con 0x7feefc0fcf80 2024-03-19T03:13:01.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.438+0000 7feef8ff9700 1 -- 172.21.15.59:0/2217939978 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7feee4026440 con 0x7feefc0fcf80 2024-03-19T03:13:01.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.438+0000 7feef8ff9700 1 --2- 172.21.15.59:0/2217939978 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7feee80692a0 0x7feee806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:01.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.438+0000 7feef8ff9700 1 -- 172.21.15.59:0/2217939978 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7feee4087ba0 con 0x7feefc0fcf80 2024-03-19T03:13:01.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.438+0000 7feefbfff700 1 --2- 172.21.15.59:0/2217939978 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7feee80692a0 0x7feee806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:01.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.439+0000 7feefbfff700 1 --2- 172.21.15.59:0/2217939978 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7feee80692a0 0x7feee806b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7feeec0041d0 tx=0x7feeec0043c0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:01.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.443+0000 7feef8ff9700 1 -- 172.21.15.59:0/2217939978 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7feee4057d10 con 0x7feefc0fcf80 2024-03-19T03:13:01.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:01 smithi176 ceph-mon[36450]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:01.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:13:01.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.651+0000 7fef022a3700 1 -- 172.21.15.59:0/2217939978 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7feefc1048d0 con 0x7feee80692a0 2024-03-19T03:13:01.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.652+0000 7feef8ff9700 1 -- 172.21.15.59:0/2217939978 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7feefc1048d0 con 0x7feee80692a0 2024-03-19T03:13:01.654 INFO:teuthology.orchestra.run.smithi059.stdout:Backend: cephadm 2024-03-19T03:13:01.654 INFO:teuthology.orchestra.run.smithi059.stdout:Available: Yes 2024-03-19T03:13:01.654 INFO:teuthology.orchestra.run.smithi059.stdout:Paused: No 2024-03-19T03:13:01.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.654+0000 7fef022a3700 1 -- 172.21.15.59:0/2217939978 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7feee80692a0 msgr2=0x7feee806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:01.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.654+0000 7fef022a3700 1 --2- 172.21.15.59:0/2217939978 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7feee80692a0 0x7feee806b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7feeec0041d0 tx=0x7feeec0043c0 comp rx=0 tx=0).stop 2024-03-19T03:13:01.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.654+0000 7fef022a3700 1 -- 172.21.15.59:0/2217939978 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feefc0fcf80 msgr2=0x7feefc10df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:01.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.654+0000 7fef022a3700 1 --2- 172.21.15.59:0/2217939978 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feefc0fcf80 0x7feefc10df50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7feee400b3b0 tx=0x7feee4009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:01.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.655+0000 7fef022a3700 1 -- 172.21.15.59:0/2217939978 shutdown_connections 2024-03-19T03:13:01.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.655+0000 7fef022a3700 1 --2- 172.21.15.59:0/2217939978 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7feee80692a0 0x7feee806b750 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:01.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.655+0000 7fef022a3700 1 --2- 172.21.15.59:0/2217939978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feefc0fbcd0 0x7feefc10da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:01.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.655+0000 7fef022a3700 1 --2- 172.21.15.59:0/2217939978 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feefc0fcf80 0x7feefc10df50 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:01.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.655+0000 7fef022a3700 1 -- 172.21.15.59:0/2217939978 >> 172.21.15.59:0/2217939978 conn(0x7feefc0f7430 msgr2=0x7feefc065b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:01.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.655+0000 7fef022a3700 1 -- 172.21.15.59:0/2217939978 shutdown_connections 2024-03-19T03:13:01.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:01.655+0000 7fef022a3700 1 -- 172.21.15.59:0/2217939978 wait complete. 2024-03-19T03:13:02.314 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-19T03:13:02.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:02 smithi059 ceph-mon[29133]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:13:02.575 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:13:02.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:02 smithi176 ceph-mon[36450]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:13:03.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:03 smithi059 ceph-mon[29133]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:03.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:03 smithi176 ceph-mon[36450]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:04.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.130+0000 7f2898f08700 1 -- 172.21.15.59:0/117033929 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28940fbcd0 msgr2=0x7f28940fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:04.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.130+0000 7f2898f08700 1 --2- 172.21.15.59:0/117033929 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28940fbcd0 0x7f28940fc0e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2888004940 tx=0x7f2888009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:04.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.130+0000 7f2898f08700 1 -- 172.21.15.59:0/117033929 shutdown_connections 2024-03-19T03:13:04.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.130+0000 7f2898f08700 1 --2- 172.21.15.59:0/117033929 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28940fcf80 0x7f28940fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:04.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.130+0000 7f2898f08700 1 --2- 172.21.15.59:0/117033929 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28940fbcd0 0x7f28940fc0e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:04.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.131+0000 7f2898f08700 1 -- 172.21.15.59:0/117033929 >> 172.21.15.59:0/117033929 conn(0x7f28940f7450 msgr2=0x7f28940f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:04.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.131+0000 7f2898f08700 1 -- 172.21.15.59:0/117033929 shutdown_connections 2024-03-19T03:13:04.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.131+0000 7f2898f08700 1 -- 172.21.15.59:0/117033929 wait complete. 2024-03-19T03:13:04.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.131+0000 7f2898f08700 1 Processor -- start 2024-03-19T03:13:04.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.131+0000 7f2898f08700 1 -- start start 2024-03-19T03:13:04.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.132+0000 7f2898f08700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28940fbcd0 0x7f289410c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:04.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.132+0000 7f2898f08700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28940fcf80 0x7f289410cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:04.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.132+0000 7f2898f08700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289410d360 con 0x7f28940fbcd0 2024-03-19T03:13:04.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.132+0000 7f2898f08700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289410d4d0 con 0x7f28940fcf80 2024-03-19T03:13:04.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.132+0000 7f289259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28940fbcd0 0x7f289410c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:04.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.132+0000 7f289259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28940fbcd0 0x7f289410c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34690/0 (socket says 172.21.15.59:34690) 2024-03-19T03:13:04.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.132+0000 7f289259c700 1 -- 172.21.15.59:0/3388902356 learned_addr learned my addr 172.21.15.59:0/3388902356 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:13:04.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.132+0000 7f2891d9b700 1 --2- 172.21.15.59:0/3388902356 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28940fcf80 0x7f289410cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:04.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.132+0000 7f289259c700 1 -- 172.21.15.59:0/3388902356 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28940fcf80 msgr2=0x7f289410cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:04.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.132+0000 7f289259c700 1 --2- 172.21.15.59:0/3388902356 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28940fcf80 0x7f289410cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:04.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.132+0000 7f289259c700 1 -- 172.21.15.59:0/3388902356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2888005040 con 0x7f28940fbcd0 2024-03-19T03:13:04.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.132+0000 7f289259c700 1 --2- 172.21.15.59:0/3388902356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28940fbcd0 0x7f289410c820 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2888009fd0 tx=0x7f2888009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:04.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.132+0000 7f2882ffd700 1 -- 172.21.15.59:0/3388902356 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f288800e040 con 0x7f28940fbcd0 2024-03-19T03:13:04.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.132+0000 7f2882ffd700 1 -- 172.21.15.59:0/3388902356 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f288800be00 con 0x7f28940fbcd0 2024-03-19T03:13:04.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.132+0000 7f2898f08700 1 -- 172.21.15.59:0/3388902356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2894066270 con 0x7f28940fbcd0 2024-03-19T03:13:04.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.132+0000 7f2882ffd700 1 -- 172.21.15.59:0/3388902356 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28880137f0 con 0x7f28940fbcd0 2024-03-19T03:13:04.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.132+0000 7f2898f08700 1 -- 172.21.15.59:0/3388902356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2894066830 con 0x7f28940fbcd0 2024-03-19T03:13:04.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.133+0000 7f2898f08700 1 -- 172.21.15.59:0/3388902356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f289404c710 con 0x7f28940fbcd0 2024-03-19T03:13:04.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.134+0000 7f2882ffd700 1 -- 172.21.15.59:0/3388902356 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f2888013950 con 0x7f28940fbcd0 2024-03-19T03:13:04.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.134+0000 7f2882ffd700 1 --2- 172.21.15.59:0/3388902356 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f287c0692a0 0x7f287c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:04.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.134+0000 7f2882ffd700 1 -- 172.21.15.59:0/3388902356 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f288808a6f0 con 0x7f28940fbcd0 2024-03-19T03:13:04.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.134+0000 7f2891d9b700 1 --2- 172.21.15.59:0/3388902356 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f287c0692a0 0x7f287c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:04.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.135+0000 7f2891d9b700 1 --2- 172.21.15.59:0/3388902356 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f287c0692a0 0x7f287c06b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2884004f40 tx=0x7f288400b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:04.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.139+0000 7f2882ffd700 1 -- 172.21.15.59:0/3388902356 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f288805a8a0 con 0x7f28940fbcd0 2024-03-19T03:13:04.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.341+0000 7f2898f08700 1 -- 172.21.15.59:0/3388902356 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2894066fd0 con 0x7f287c0692a0 2024-03-19T03:13:04.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.351+0000 7f2882ffd700 1 -- 172.21.15.59:0/3388902356 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2682 (secure 0 0 0) 0x7f2894066fd0 con 0x7f287c0692a0 2024-03-19T03:13:04.353 INFO:teuthology.orchestra.run.smithi059.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-19T03:13:04.353 INFO:teuthology.orchestra.run.smithi059.stdout:alertmanager.smithi059 smithi059 *:9093,9094 running (3m) 2m ago 5m 26.5M - 0.25.0 c8568f914cd2 40a81ac05a90 2024-03-19T03:13:04.353 INFO:teuthology.orchestra.run.smithi059.stdout:crash.smithi059 smithi059 running (5m) 2m ago 5m 6669k - 17.2.7-700-gb4843bd7 810955603526 3090e46c4bff 2024-03-19T03:13:04.353 INFO:teuthology.orchestra.run.smithi059.stdout:crash.smithi176 smithi176 running (4m) 67s ago 4m 6660k - 17.2.7-700-gb4843bd7 810955603526 2df9a0e61e4c 2024-03-19T03:13:04.353 INFO:teuthology.orchestra.run.smithi059.stdout:grafana.smithi059 smithi059 *:3000 running (3m) 2m ago 4m 78.3M - 9.4.7 954c08fa6188 b3394e3d4026 2024-03-19T03:13:04.353 INFO:teuthology.orchestra.run.smithi059.stdout:mgr.smithi059.azpecc smithi059 *:9283 running (6m) 2m ago 6m 470M - 17.2.7-700-gb4843bd7 810955603526 441ca0cfcfac 2024-03-19T03:13:04.353 INFO:teuthology.orchestra.run.smithi059.stdout:mgr.smithi176.ycmqbw smithi176 *:8443,9283 running (4m) 67s ago 4m 432M - 17.2.7-700-gb4843bd7 810955603526 ac8bc9c6ec9b 2024-03-19T03:13:04.353 INFO:teuthology.orchestra.run.smithi059.stdout:mon.smithi059 smithi059 running (6m) 2m ago 6m 44.6M 2048M 17.2.7-700-gb4843bd7 810955603526 c8bc018eaa37 2024-03-19T03:13:04.353 INFO:teuthology.orchestra.run.smithi059.stdout:mon.smithi176 smithi176 running (4m) 67s ago 4m 38.9M 2048M 17.2.7-700-gb4843bd7 810955603526 4130cce6243b 2024-03-19T03:13:04.354 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter.smithi059 smithi059 *:9100 running (5m) 2m ago 5m 16.2M - 1.5.0 0da6a335fe13 d3fca5e9a667 2024-03-19T03:13:04.354 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter.smithi176 smithi176 *:9100 running (4m) 67s ago 4m 14.6M - 1.5.0 0da6a335fe13 35389a3c33fd 2024-03-19T03:13:04.354 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 smithi059 running (3m) 2m ago 3m 35.8M 4289M 17.2.7-700-gb4843bd7 810955603526 2511279e8852 2024-03-19T03:13:04.354 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 smithi059 running (2m) 2m ago 2m 36.0M 4289M 17.2.7-700-gb4843bd7 810955603526 cd85f348d650 2024-03-19T03:13:04.354 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 smithi059 running (2m) 2m ago 2m 38.7M 4289M 17.2.7-700-gb4843bd7 810955603526 363582327fde 2024-03-19T03:13:04.354 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 smithi176 running (2m) 67s ago 2m 38.6M 5312M 17.2.7-700-gb4843bd7 810955603526 436c587b5265 2024-03-19T03:13:04.354 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 smithi176 running (98s) 67s ago 98s 35.9M 5312M 17.2.7-700-gb4843bd7 810955603526 80d2dd2cd18b 2024-03-19T03:13:04.354 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 smithi176 running (72s) 67s ago 72s 10.1M 5312M 17.2.7-700-gb4843bd7 810955603526 dd0f98b5ec05 2024-03-19T03:13:04.354 INFO:teuthology.orchestra.run.smithi059.stdout:prometheus.smithi059 smithi059 *:9095 running (3m) 2m ago 4m 36.0M - 2.43.0 a07b618ecd1d 1782702019d3 2024-03-19T03:13:04.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.353+0000 7f2898f08700 1 -- 172.21.15.59:0/3388902356 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f287c0692a0 msgr2=0x7f287c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:04.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.353+0000 7f2898f08700 1 --2- 172.21.15.59:0/3388902356 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f287c0692a0 0x7f287c06b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2884004f40 tx=0x7f288400b040 comp rx=0 tx=0).stop 2024-03-19T03:13:04.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.353+0000 7f2898f08700 1 -- 172.21.15.59:0/3388902356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28940fbcd0 msgr2=0x7f289410c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:04.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.353+0000 7f2898f08700 1 --2- 172.21.15.59:0/3388902356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28940fbcd0 0x7f289410c820 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2888009fd0 tx=0x7f2888009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:04.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.353+0000 7f2898f08700 1 -- 172.21.15.59:0/3388902356 shutdown_connections 2024-03-19T03:13:04.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.353+0000 7f2898f08700 1 --2- 172.21.15.59:0/3388902356 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f287c0692a0 0x7f287c06b750 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:04.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.353+0000 7f2898f08700 1 --2- 172.21.15.59:0/3388902356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28940fbcd0 0x7f289410c820 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:04.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.353+0000 7f2898f08700 1 --2- 172.21.15.59:0/3388902356 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28940fcf80 0x7f289410cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:04.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.353+0000 7f2898f08700 1 -- 172.21.15.59:0/3388902356 >> 172.21.15.59:0/3388902356 conn(0x7f28940f7450 msgr2=0x7f28940f8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:04.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.353+0000 7f2898f08700 1 -- 172.21.15.59:0/3388902356 shutdown_connections 2024-03-19T03:13:04.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:04.353+0000 7f2898f08700 1 -- 172.21.15.59:0/3388902356 wait complete. 2024-03-19T03:13:05.029 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-19T03:13:05.334 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:13:06.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:06 smithi059 ceph-mon[29133]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:13:06.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:06 smithi059 ceph-mon[29133]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:06.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:06 smithi176 ceph-mon[36450]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:13:06.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:06 smithi176 ceph-mon[36450]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:06.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.900+0000 7fe75a2c0700 1 -- 172.21.15.59:0/2612948411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7540fe040 msgr2=0x7fe7540fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:06.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.900+0000 7fe75a2c0700 1 --2- 172.21.15.59:0/2612948411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7540fe040 0x7fe7540fe4b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe73c004940 tx=0x7fe73c009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:06.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.900+0000 7fe75a2c0700 1 -- 172.21.15.59:0/2612948411 shutdown_connections 2024-03-19T03:13:06.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.900+0000 7fe75a2c0700 1 --2- 172.21.15.59:0/2612948411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7540fe040 0x7fe7540fe4b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:06.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.900+0000 7fe75a2c0700 1 --2- 172.21.15.59:0/2612948411 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7540fce40 0x7fe7540fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:06.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.900+0000 7fe75a2c0700 1 -- 172.21.15.59:0/2612948411 >> 172.21.15.59:0/2612948411 conn(0x7fe7540f8550 msgr2=0x7fe7540fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:06.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.900+0000 7fe75a2c0700 1 -- 172.21.15.59:0/2612948411 shutdown_connections 2024-03-19T03:13:06.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.900+0000 7fe75a2c0700 1 -- 172.21.15.59:0/2612948411 wait complete. 2024-03-19T03:13:06.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.900+0000 7fe75a2c0700 1 Processor -- start 2024-03-19T03:13:06.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.900+0000 7fe75a2c0700 1 -- start start 2024-03-19T03:13:06.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.901+0000 7fe75a2c0700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7540fce40 0x7fe75410d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:06.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.901+0000 7fe75a2c0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7540fe040 0x7fe75410ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:06.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.901+0000 7fe75a2c0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75410e4d0 con 0x7fe7540fe040 2024-03-19T03:13:06.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.901+0000 7fe75a2c0700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75410e640 con 0x7fe7540fce40 2024-03-19T03:13:06.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.901+0000 7fe74bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7540fe040 0x7fe75410ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:06.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.901+0000 7fe74bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7540fe040 0x7fe75410ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34704/0 (socket says 172.21.15.59:34704) 2024-03-19T03:13:06.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.901+0000 7fe74bfff700 1 -- 172.21.15.59:0/1502415335 learned_addr learned my addr 172.21.15.59:0/1502415335 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:13:06.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.901+0000 7fe753fff700 1 --2- 172.21.15.59:0/1502415335 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7540fce40 0x7fe75410d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:06.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.901+0000 7fe74bfff700 1 -- 172.21.15.59:0/1502415335 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7540fce40 msgr2=0x7fe75410d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:06.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.901+0000 7fe74bfff700 1 --2- 172.21.15.59:0/1502415335 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7540fce40 0x7fe75410d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:06.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.901+0000 7fe74bfff700 1 -- 172.21.15.59:0/1502415335 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73c005040 con 0x7fe7540fe040 2024-03-19T03:13:06.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.901+0000 7fe74bfff700 1 --2- 172.21.15.59:0/1502415335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7540fe040 0x7fe75410ded0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00b3b0 tx=0x7fe73c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:06.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.901+0000 7fe7517fa700 1 -- 172.21.15.59:0/1502415335 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe73c00e040 con 0x7fe7540fe040 2024-03-19T03:13:06.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.901+0000 7fe7517fa700 1 -- 172.21.15.59:0/1502415335 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe73c00bcc0 con 0x7fe7540fe040 2024-03-19T03:13:06.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.901+0000 7fe75a2c0700 1 -- 172.21.15.59:0/1502415335 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe754112fa0 con 0x7fe7540fe040 2024-03-19T03:13:06.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.901+0000 7fe7517fa700 1 -- 172.21.15.59:0/1502415335 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe73c01c630 con 0x7fe7540fe040 2024-03-19T03:13:06.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.901+0000 7fe75a2c0700 1 -- 172.21.15.59:0/1502415335 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe754113530 con 0x7fe7540fe040 2024-03-19T03:13:06.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.902+0000 7fe75a2c0700 1 -- 172.21.15.59:0/1502415335 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe75404c710 con 0x7fe7540fe040 2024-03-19T03:13:06.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.903+0000 7fe7517fa700 1 -- 172.21.15.59:0/1502415335 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe73c013980 con 0x7fe7540fe040 2024-03-19T03:13:06.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.903+0000 7fe7517fa700 1 --2- 172.21.15.59:0/1502415335 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe7400692a0 0x7fe74006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:06.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.903+0000 7fe7517fa700 1 -- 172.21.15.59:0/1502415335 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe73c087b90 con 0x7fe7540fe040 2024-03-19T03:13:06.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.903+0000 7fe753fff700 1 --2- 172.21.15.59:0/1502415335 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe7400692a0 0x7fe74006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:06.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.904+0000 7fe753fff700 1 --2- 172.21.15.59:0/1502415335 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe7400692a0 0x7fe74006b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe744009fd0 tx=0x7fe744009f00 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:06.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:06.908+0000 7fe7517fa700 1 -- 172.21.15.59:0/1502415335 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fe73c057c90 con 0x7fe7540fe040 2024-03-19T03:13:07.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:07.111+0000 7fe75a2c0700 1 -- 172.21.15.59:0/1502415335 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe754113770 con 0x7fe7400692a0 2024-03-19T03:13:07.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:07.115+0000 7fe7517fa700 1 -- 172.21.15.59:0/1502415335 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1170 (secure 0 0 0) 0x7fe754113770 con 0x7fe7400692a0 2024-03-19T03:13:07.117 INFO:teuthology.orchestra.run.smithi059.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-19T03:13:07.117 INFO:teuthology.orchestra.run.smithi059.stdout:alertmanager ?:9093,9094 1/1 2m ago 5m count:1 2024-03-19T03:13:07.117 INFO:teuthology.orchestra.run.smithi059.stdout:crash 2/2 2m ago 6m * 2024-03-19T03:13:07.117 INFO:teuthology.orchestra.run.smithi059.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-03-19T03:13:07.117 INFO:teuthology.orchestra.run.smithi059.stdout:mgr 2/2 2m ago 6m count:2 2024-03-19T03:13:07.117 INFO:teuthology.orchestra.run.smithi059.stdout:mon 2/2 2m ago 5m smithi059:172.21.15.59=smithi059;smithi176:172.21.15.176=smithi176;count:2 2024-03-19T03:13:07.117 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter ?:9100 2/2 2m ago 5m * 2024-03-19T03:13:07.117 INFO:teuthology.orchestra.run.smithi059.stdout:osd 6 2m ago - 2024-03-19T03:13:07.117 INFO:teuthology.orchestra.run.smithi059.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-03-19T03:13:07.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:07.117+0000 7fe75a2c0700 1 -- 172.21.15.59:0/1502415335 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe7400692a0 msgr2=0x7fe74006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:07.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:07.117+0000 7fe75a2c0700 1 --2- 172.21.15.59:0/1502415335 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe7400692a0 0x7fe74006b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe744009fd0 tx=0x7fe744009f00 comp rx=0 tx=0).stop 2024-03-19T03:13:07.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:07.117+0000 7fe75a2c0700 1 -- 172.21.15.59:0/1502415335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7540fe040 msgr2=0x7fe75410ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:07.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:07.117+0000 7fe75a2c0700 1 --2- 172.21.15.59:0/1502415335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7540fe040 0x7fe75410ded0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00b3b0 tx=0x7fe73c009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:07.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:07.117+0000 7fe75a2c0700 1 -- 172.21.15.59:0/1502415335 shutdown_connections 2024-03-19T03:13:07.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:07.117+0000 7fe75a2c0700 1 --2- 172.21.15.59:0/1502415335 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe7400692a0 0x7fe74006b750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:07.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:07.117+0000 7fe75a2c0700 1 --2- 172.21.15.59:0/1502415335 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7540fce40 0x7fe75410d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:07.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:07.117+0000 7fe75a2c0700 1 --2- 172.21.15.59:0/1502415335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7540fe040 0x7fe75410ded0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:07.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:07.117+0000 7fe75a2c0700 1 -- 172.21.15.59:0/1502415335 >> 172.21.15.59:0/1502415335 conn(0x7fe7540f8550 msgr2=0x7fe7540f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:07.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:07.117+0000 7fe75a2c0700 1 -- 172.21.15.59:0/1502415335 shutdown_connections 2024-03-19T03:13:07.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:07.117+0000 7fe75a2c0700 1 -- 172.21.15.59:0/1502415335 wait complete. 2024-03-19T03:13:07.647 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-19T03:13:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:13:08.334 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:08 smithi059 ceph-mon[29133]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:08.334 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:08 smithi059 ceph-mon[29133]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:13:08.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:08 smithi176 ceph-mon[36450]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:08.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:08 smithi176 ceph-mon[36450]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:13:09.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.426+0000 7f8dee884700 1 -- 172.21.15.59:0/155875804 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8de80fed80 msgr2=0x7f8de80faec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:09.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.426+0000 7f8dee884700 1 --2- 172.21.15.59:0/155875804 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8de80fed80 0x7f8de80faec0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc004940 tx=0x7f8ddc009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:09.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.426+0000 7f8dee884700 1 -- 172.21.15.59:0/155875804 shutdown_connections 2024-03-19T03:13:09.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.426+0000 7f8dee884700 1 --2- 172.21.15.59:0/155875804 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8de80fed80 0x7f8de80faec0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:09.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.426+0000 7f8dee884700 1 --2- 172.21.15.59:0/155875804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8de80fe430 0x7f8de80fe840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:09.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.426+0000 7f8dee884700 1 -- 172.21.15.59:0/155875804 >> 172.21.15.59:0/155875804 conn(0x7f8de80f6bd0 msgr2=0x7f8de80f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:09.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.426+0000 7f8dee884700 1 -- 172.21.15.59:0/155875804 shutdown_connections 2024-03-19T03:13:09.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.426+0000 7f8dee884700 1 -- 172.21.15.59:0/155875804 wait complete. 2024-03-19T03:13:09.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.427+0000 7f8dee884700 1 Processor -- start 2024-03-19T03:13:09.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.427+0000 7f8dee884700 1 -- start start 2024-03-19T03:13:09.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.427+0000 7f8dee884700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8de80fe430 0x7f8de810d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:09.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.427+0000 7f8dee884700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8de80fed80 0x7f8de810de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:09.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.427+0000 7f8dee884700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de810e440 con 0x7f8de80fed80 2024-03-19T03:13:09.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.427+0000 7f8dee884700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de810e5b0 con 0x7f8de80fe430 2024-03-19T03:13:09.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.427+0000 7f8de77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8de80fed80 0x7f8de810de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:09.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.427+0000 7f8de77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8de80fed80 0x7f8de810de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34716/0 (socket says 172.21.15.59:34716) 2024-03-19T03:13:09.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.427+0000 7f8de77fe700 1 -- 172.21.15.59:0/2750842077 learned_addr learned my addr 172.21.15.59:0/2750842077 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:13:09.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.427+0000 7f8de7fff700 1 --2- 172.21.15.59:0/2750842077 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8de80fe430 0x7f8de810d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:09.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.427+0000 7f8de77fe700 1 -- 172.21.15.59:0/2750842077 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8de80fe430 msgr2=0x7f8de810d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:09.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.427+0000 7f8de77fe700 1 --2- 172.21.15.59:0/2750842077 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8de80fe430 0x7f8de810d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:09.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.427+0000 7f8de77fe700 1 -- 172.21.15.59:0/2750842077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ddc005040 con 0x7f8de80fed80 2024-03-19T03:13:09.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.428+0000 7f8de77fe700 1 --2- 172.21.15.59:0/2750842077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8de80fed80 0x7f8de810de40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc009150 tx=0x7f8ddc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:09.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.428+0000 7f8de4ff9700 1 -- 172.21.15.59:0/2750842077 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ddc00e040 con 0x7f8de80fed80 2024-03-19T03:13:09.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.428+0000 7f8de4ff9700 1 -- 172.21.15.59:0/2750842077 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ddc004290 con 0x7f8de80fed80 2024-03-19T03:13:09.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.428+0000 7f8dee884700 1 -- 172.21.15.59:0/2750842077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de8112fb0 con 0x7f8de80fed80 2024-03-19T03:13:09.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.428+0000 7f8de4ff9700 1 -- 172.21.15.59:0/2750842077 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ddc01c630 con 0x7f8de80fed80 2024-03-19T03:13:09.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.428+0000 7f8dee884700 1 -- 172.21.15.59:0/2750842077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de8113540 con 0x7f8de80fed80 2024-03-19T03:13:09.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.429+0000 7f8dee884700 1 -- 172.21.15.59:0/2750842077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de804c710 con 0x7f8de80fed80 2024-03-19T03:13:09.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.429+0000 7f8de4ff9700 1 -- 172.21.15.59:0/2750842077 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f8ddc00bbf0 con 0x7f8de80fed80 2024-03-19T03:13:09.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.430+0000 7f8de4ff9700 1 --2- 172.21.15.59:0/2750842077 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f8dd00692a0 0x7f8dd006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:09.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.430+0000 7f8de4ff9700 1 -- 172.21.15.59:0/2750842077 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8ddc087a00 con 0x7f8de80fed80 2024-03-19T03:13:09.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.430+0000 7f8de7fff700 1 --2- 172.21.15.59:0/2750842077 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f8dd00692a0 0x7f8dd006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:09.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.430+0000 7f8de7fff700 1 --2- 172.21.15.59:0/2750842077 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f8dd00692a0 0x7f8dd006b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8004f40 tx=0x7f8dd800b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:09.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.434+0000 7f8de4ff9700 1 -- 172.21.15.59:0/2750842077 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f8ddc057ba0 con 0x7f8de80fed80 2024-03-19T03:13:09.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.634+0000 7f8dee884700 1 -- 172.21.15.59:0/2750842077 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8de81024d0 con 0x7f8dd00692a0 2024-03-19T03:13:09.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.635+0000 7f8de4ff9700 1 -- 172.21.15.59:0/2750842077 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f8de81024d0 con 0x7f8dd00692a0 2024-03-19T03:13:09.637 INFO:teuthology.orchestra.run.smithi059.stdout:HOST ADDR LABELS STATUS 2024-03-19T03:13:09.637 INFO:teuthology.orchestra.run.smithi059.stdout:smithi059 172.21.15.59 2024-03-19T03:13:09.637 INFO:teuthology.orchestra.run.smithi059.stdout:smithi176 172.21.15.176 2024-03-19T03:13:09.637 INFO:teuthology.orchestra.run.smithi059.stdout:2 hosts in cluster 2024-03-19T03:13:09.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.637+0000 7f8dee884700 1 -- 172.21.15.59:0/2750842077 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f8dd00692a0 msgr2=0x7f8dd006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:09.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.637+0000 7f8dee884700 1 --2- 172.21.15.59:0/2750842077 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f8dd00692a0 0x7f8dd006b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8004f40 tx=0x7f8dd800b040 comp rx=0 tx=0).stop 2024-03-19T03:13:09.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.637+0000 7f8dee884700 1 -- 172.21.15.59:0/2750842077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8de80fed80 msgr2=0x7f8de810de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:09.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.637+0000 7f8dee884700 1 --2- 172.21.15.59:0/2750842077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8de80fed80 0x7f8de810de40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc009150 tx=0x7f8ddc009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:09.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.638+0000 7f8dee884700 1 -- 172.21.15.59:0/2750842077 shutdown_connections 2024-03-19T03:13:09.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.638+0000 7f8dee884700 1 --2- 172.21.15.59:0/2750842077 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f8dd00692a0 0x7f8dd006b750 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:09.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.638+0000 7f8dee884700 1 --2- 172.21.15.59:0/2750842077 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8de80fe430 0x7f8de810d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:09.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.638+0000 7f8dee884700 1 --2- 172.21.15.59:0/2750842077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8de80fed80 0x7f8de810de40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:09.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.638+0000 7f8dee884700 1 -- 172.21.15.59:0/2750842077 >> 172.21.15.59:0/2750842077 conn(0x7f8de80f6bd0 msgr2=0x7f8de80f8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:09.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.638+0000 7f8dee884700 1 -- 172.21.15.59:0/2750842077 shutdown_connections 2024-03-19T03:13:09.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:09.638+0000 7f8dee884700 1 -- 172.21.15.59:0/2750842077 wait complete. 2024-03-19T03:13:10.227 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:10 smithi059 ceph-mon[29133]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:10.253 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-19T03:13:10.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:10 smithi176 ceph-mon[36450]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:10.513 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:13:11.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:11 smithi059 ceph-mon[29133]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:13:11.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:11 smithi176 ceph-mon[36450]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:13:12.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.181+0000 7f340f0a4700 1 -- 172.21.15.59:0/1589186521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34080fcf90 msgr2=0x7f34080fd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:12.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.181+0000 7f340f0a4700 1 --2- 172.21.15.59:0/1589186521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34080fcf90 0x7f34080fd400 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f33f4004940 tx=0x7f33f4009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:12.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.181+0000 7f340f0a4700 1 -- 172.21.15.59:0/1589186521 shutdown_connections 2024-03-19T03:13:12.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.181+0000 7f340f0a4700 1 --2- 172.21.15.59:0/1589186521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34080fcf90 0x7f34080fd400 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:12.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.181+0000 7f340f0a4700 1 --2- 172.21.15.59:0/1589186521 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f34080fbce0 0x7f34080fc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:12.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.181+0000 7f340f0a4700 1 -- 172.21.15.59:0/1589186521 >> 172.21.15.59:0/1589186521 conn(0x7f34080f7440 msgr2=0x7f34080f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:12.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.181+0000 7f340f0a4700 1 -- 172.21.15.59:0/1589186521 shutdown_connections 2024-03-19T03:13:12.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.181+0000 7f340f0a4700 1 -- 172.21.15.59:0/1589186521 wait complete. 2024-03-19T03:13:12.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.182+0000 7f340f0a4700 1 Processor -- start 2024-03-19T03:13:12.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.182+0000 7f340f0a4700 1 -- start start 2024-03-19T03:13:12.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.182+0000 7f340f0a4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34080fbce0 0x7f3408066d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:12.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.182+0000 7f340f0a4700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f34080fcf90 0x7f3408112760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:12.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.182+0000 7f340f0a4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3408112ca0 con 0x7f34080fbce0 2024-03-19T03:13:12.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.182+0000 7f340f0a4700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3408112de0 con 0x7f34080fcf90 2024-03-19T03:13:12.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.182+0000 7f340ce40700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34080fbce0 0x7f3408066d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:12.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.182+0000 7f340ce40700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34080fbce0 0x7f3408066d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34730/0 (socket says 172.21.15.59:34730) 2024-03-19T03:13:12.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.182+0000 7f340ce40700 1 -- 172.21.15.59:0/2064575687 learned_addr learned my addr 172.21.15.59:0/2064575687 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:13:12.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.183+0000 7f3407fff700 1 --2- 172.21.15.59:0/2064575687 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f34080fcf90 0x7f3408112760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:12.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.183+0000 7f340ce40700 1 -- 172.21.15.59:0/2064575687 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f34080fcf90 msgr2=0x7f3408112760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:12.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.183+0000 7f340ce40700 1 --2- 172.21.15.59:0/2064575687 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f34080fcf90 0x7f3408112760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:12.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.183+0000 7f340ce40700 1 -- 172.21.15.59:0/2064575687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f4005040 con 0x7f34080fbce0 2024-03-19T03:13:12.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.183+0000 7f340ce40700 1 --2- 172.21.15.59:0/2064575687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34080fbce0 0x7f3408066d50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f33fc002a50 tx=0x7f33fc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:12.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.183+0000 7f34057fa700 1 -- 172.21.15.59:0/2064575687 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33fc007cc0 con 0x7f34080fbce0 2024-03-19T03:13:12.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.183+0000 7f34057fa700 1 -- 172.21.15.59:0/2064575687 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f33fc00f040 con 0x7f34080fbce0 2024-03-19T03:13:12.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.183+0000 7f340f0a4700 1 -- 172.21.15.59:0/2064575687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3408112fe0 con 0x7f34080fbce0 2024-03-19T03:13:12.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.183+0000 7f34057fa700 1 -- 172.21.15.59:0/2064575687 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33fc00db70 con 0x7f34080fbce0 2024-03-19T03:13:12.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.183+0000 7f340f0a4700 1 -- 172.21.15.59:0/2064575687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3408113570 con 0x7f34080fbce0 2024-03-19T03:13:12.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.184+0000 7f340f0a4700 1 -- 172.21.15.59:0/2064575687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f340804c710 con 0x7f34080fbce0 2024-03-19T03:13:12.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.184+0000 7f34057fa700 1 -- 172.21.15.59:0/2064575687 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f33fc00dd10 con 0x7f34080fbce0 2024-03-19T03:13:12.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.185+0000 7f34057fa700 1 --2- 172.21.15.59:0/2064575687 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f33f80692a0 0x7f33f806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:12.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.185+0000 7f34057fa700 1 -- 172.21.15.59:0/2064575687 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f33fc086580 con 0x7f34080fbce0 2024-03-19T03:13:12.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.185+0000 7f3407fff700 1 --2- 172.21.15.59:0/2064575687 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f33f80692a0 0x7f33f806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:12.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.185+0000 7f3407fff700 1 --2- 172.21.15.59:0/2064575687 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f33f80692a0 0x7f33f806b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f33f4000c00 tx=0x7f33f4009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:12.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.190+0000 7f34057fa700 1 -- 172.21.15.59:0/2064575687 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f33fc056600 con 0x7f34080fbce0 2024-03-19T03:13:12.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:12 smithi059 ceph-mon[29133]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:12.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:12 smithi176 ceph-mon[36450]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:12.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.392+0000 7f340f0a4700 1 -- 172.21.15.59:0/2064575687 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3408113790 con 0x7f33f80692a0 2024-03-19T03:13:12.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.394+0000 7f34057fa700 1 -- 172.21.15.59:0/2064575687 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f3408113790 con 0x7f33f80692a0 2024-03-19T03:13:12.395 INFO:teuthology.orchestra.run.smithi059.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-19T03:13:12.395 INFO:teuthology.orchestra.run.smithi059.stdout:smithi059 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330009L400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-19T03:13:12.395 INFO:teuthology.orchestra.run.smithi059.stdout:smithi176 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT62330099400BGN 372G No 71s ago Has a FileSystem, LVM detected 2024-03-19T03:13:12.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.395+0000 7f340f0a4700 1 -- 172.21.15.59:0/2064575687 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f33f80692a0 msgr2=0x7f33f806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:12.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.395+0000 7f340f0a4700 1 --2- 172.21.15.59:0/2064575687 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f33f80692a0 0x7f33f806b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f33f4000c00 tx=0x7f33f4009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:12.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.395+0000 7f340f0a4700 1 -- 172.21.15.59:0/2064575687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34080fbce0 msgr2=0x7f3408066d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:12.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.395+0000 7f340f0a4700 1 --2- 172.21.15.59:0/2064575687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34080fbce0 0x7f3408066d50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f33fc002a50 tx=0x7f33fc0046b0 comp rx=0 tx=0).stop 2024-03-19T03:13:12.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.396+0000 7f340f0a4700 1 -- 172.21.15.59:0/2064575687 shutdown_connections 2024-03-19T03:13:12.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.396+0000 7f340f0a4700 1 --2- 172.21.15.59:0/2064575687 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f33f80692a0 0x7f33f806b750 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:12.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.396+0000 7f340f0a4700 1 --2- 172.21.15.59:0/2064575687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34080fbce0 0x7f3408066d50 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:12.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.396+0000 7f340f0a4700 1 --2- 172.21.15.59:0/2064575687 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f34080fcf90 0x7f3408112760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:12.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.396+0000 7f340f0a4700 1 -- 172.21.15.59:0/2064575687 >> 172.21.15.59:0/2064575687 conn(0x7f34080f7440 msgr2=0x7f34080f8b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:12.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.396+0000 7f340f0a4700 1 -- 172.21.15.59:0/2064575687 shutdown_connections 2024-03-19T03:13:12.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:12.396+0000 7f340f0a4700 1 -- 172.21.15.59:0/2064575687 wait complete. 2024-03-19T03:13:12.940 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-19T03:13:12.953 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2024-03-19T03:13:12.953 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-03-19T03:13:13.215 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:13:14.330 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:14 smithi059 ceph-mon[29133]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:13:14.330 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:14 smithi059 ceph-mon[29133]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:14.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:14 smithi176 ceph-mon[36450]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:13:14.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:14 smithi176 ceph-mon[36450]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:14.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.906+0000 7f89653b3700 1 -- 172.21.15.59:0/2811529501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89600fff00 msgr2=0x7f8960102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:14.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.906+0000 7f89653b3700 1 --2- 172.21.15.59:0/2811529501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89600fff00 0x7f8960102320 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8948004940 tx=0x7f8948009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:14.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.906+0000 7f89653b3700 1 -- 172.21.15.59:0/2811529501 shutdown_connections 2024-03-19T03:13:14.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.906+0000 7f89653b3700 1 --2- 172.21.15.59:0/2811529501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89600fff00 0x7f8960102320 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:14.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.906+0000 7f89653b3700 1 --2- 172.21.15.59:0/2811529501 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89600662e0 0x7f89600ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:14.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.906+0000 7f89653b3700 1 -- 172.21.15.59:0/2811529501 >> 172.21.15.59:0/2811529501 conn(0x7f89600f7430 msgr2=0x7f89600f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:14.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.906+0000 7f89653b3700 1 -- 172.21.15.59:0/2811529501 shutdown_connections 2024-03-19T03:13:14.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.906+0000 7f89653b3700 1 -- 172.21.15.59:0/2811529501 wait complete. 2024-03-19T03:13:14.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.906+0000 7f89653b3700 1 Processor -- start 2024-03-19T03:13:14.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.906+0000 7f89653b3700 1 -- start start 2024-03-19T03:13:14.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.906+0000 7f89653b3700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89600662e0 0x7f896010da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:14.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.906+0000 7f89653b3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89600fff00 0x7f896010df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:14.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.907+0000 7f89653b3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896010e590 con 0x7f89600fff00 2024-03-19T03:13:14.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.907+0000 7f89653b3700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896010ede0 con 0x7f89600662e0 2024-03-19T03:13:14.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.907+0000 7f8957fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89600fff00 0x7f896010df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:14.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.907+0000 7f8957fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89600fff00 0x7f896010df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51428/0 (socket says 172.21.15.59:51428) 2024-03-19T03:13:14.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.907+0000 7f8957fff700 1 -- 172.21.15.59:0/4196142762 learned_addr learned my addr 172.21.15.59:0/4196142762 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:13:14.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.907+0000 7f895effd700 1 --2- 172.21.15.59:0/4196142762 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89600662e0 0x7f896010da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:14.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.907+0000 7f8957fff700 1 -- 172.21.15.59:0/4196142762 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89600662e0 msgr2=0x7f896010da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:14.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.907+0000 7f8957fff700 1 --2- 172.21.15.59:0/4196142762 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89600662e0 0x7f896010da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:14.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.907+0000 7f8957fff700 1 -- 172.21.15.59:0/4196142762 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8948005040 con 0x7f89600fff00 2024-03-19T03:13:14.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.907+0000 7f8957fff700 1 --2- 172.21.15.59:0/4196142762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89600fff00 0x7f896010df90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8948009150 tx=0x7f8948009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:14.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.907+0000 7f89577fe700 1 -- 172.21.15.59:0/4196142762 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f894800e040 con 0x7f89600fff00 2024-03-19T03:13:14.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.907+0000 7f89577fe700 1 -- 172.21.15.59:0/4196142762 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f894800bbf0 con 0x7f89600fff00 2024-03-19T03:13:14.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.907+0000 7f89577fe700 1 -- 172.21.15.59:0/4196142762 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f894801c630 con 0x7f89600fff00 2024-03-19T03:13:14.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.907+0000 7f89653b3700 1 -- 172.21.15.59:0/4196142762 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f896010f060 con 0x7f89600fff00 2024-03-19T03:13:14.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.908+0000 7f89653b3700 1 -- 172.21.15.59:0/4196142762 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f896010f5a0 con 0x7f89600fff00 2024-03-19T03:13:14.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.909+0000 7f89653b3700 1 -- 172.21.15.59:0/4196142762 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f896005f5f0 con 0x7f89600fff00 2024-03-19T03:13:14.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.909+0000 7f89577fe700 1 -- 172.21.15.59:0/4196142762 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f894800bd60 con 0x7f89600fff00 2024-03-19T03:13:14.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.910+0000 7f89577fe700 1 --2- 172.21.15.59:0/4196142762 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f894c0692b0 0x7f894c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:14.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.910+0000 7f89577fe700 1 -- 172.21.15.59:0/4196142762 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f89480880d0 con 0x7f89600fff00 2024-03-19T03:13:14.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.910+0000 7f895effd700 1 --2- 172.21.15.59:0/4196142762 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f894c0692b0 0x7f894c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:14.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.910+0000 7f895effd700 1 --2- 172.21.15.59:0/4196142762 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f894c0692b0 0x7f894c06b760 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8960066020 tx=0x7f895000b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:14.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:14.914+0000 7f89577fe700 1 -- 172.21.15.59:0/4196142762 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f8948058320 con 0x7f89600fff00 2024-03-19T03:13:15.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:15.139+0000 7f89653b3700 1 -- 172.21.15.59:0/4196142762 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f896010f8d0 con 0x7f894c0692b0 2024-03-19T03:13:16.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:16 smithi059 ceph-mon[29133]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:16.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:16 smithi059 ceph-mon[29133]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:13:16.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:16 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-19T03:13:16.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:16 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:13:16.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:16 smithi176 ceph-mon[36450]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:16.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:16 smithi176 ceph-mon[36450]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:13:16.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:16 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-19T03:13:16.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:16 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:13:17.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:17.061+0000 7f89577fe700 1 -- 172.21.15.59:0/4196142762 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f896010f8d0 con 0x7f894c0692b0 2024-03-19T03:13:17.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:17.064+0000 7f89653b3700 1 -- 172.21.15.59:0/4196142762 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f894c0692b0 msgr2=0x7f894c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:17.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:17.064+0000 7f89653b3700 1 --2- 172.21.15.59:0/4196142762 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f894c0692b0 0x7f894c06b760 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8960066020 tx=0x7f895000b040 comp rx=0 tx=0).stop 2024-03-19T03:13:17.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:17.064+0000 7f89653b3700 1 -- 172.21.15.59:0/4196142762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89600fff00 msgr2=0x7f896010df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:17.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:17.064+0000 7f89653b3700 1 --2- 172.21.15.59:0/4196142762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89600fff00 0x7f896010df90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8948009150 tx=0x7f8948009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:17.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:17.064+0000 7f89653b3700 1 -- 172.21.15.59:0/4196142762 shutdown_connections 2024-03-19T03:13:17.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:17.064+0000 7f89653b3700 1 --2- 172.21.15.59:0/4196142762 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f894c0692b0 0x7f894c06b760 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:17.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:17.064+0000 7f89653b3700 1 --2- 172.21.15.59:0/4196142762 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89600662e0 0x7f896010da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:17.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:17.064+0000 7f89653b3700 1 --2- 172.21.15.59:0/4196142762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89600fff00 0x7f896010df90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:17.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:17.064+0000 7f89653b3700 1 -- 172.21.15.59:0/4196142762 >> 172.21.15.59:0/4196142762 conn(0x7f89600f7430 msgr2=0x7f89600f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:17.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:17.065+0000 7f89653b3700 1 -- 172.21.15.59:0/4196142762 shutdown_connections 2024-03-19T03:13:17.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:17.065+0000 7f89653b3700 1 -- 172.21.15.59:0/4196142762 wait complete. 2024-03-19T03:13:17.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:17 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-19T03:13:17.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:17 smithi059 ceph-mon[29133]: osdmap e41: 6 total, 6 up, 6 in 2024-03-19T03:13:17.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:17 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-19T03:13:17.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:13:17.038+0000 7f425c6f4700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-19T03:13:17.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:17 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-19T03:13:17.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:17 smithi176 ceph-mon[36450]: osdmap e41: 6 total, 6 up, 6 in 2024-03-19T03:13:17.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:17 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-19T03:13:17.847 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-19T03:13:18.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:18 smithi059 ceph-mon[29133]: pgmap v172: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:18.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:18 smithi059 ceph-mon[29133]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T03:13:18.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:18 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-19T03:13:18.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:18 smithi059 ceph-mon[29133]: osdmap e42: 6 total, 6 up, 6 in 2024-03-19T03:13:18.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:18 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-19T03:13:18.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:18 smithi059 ceph-mon[29133]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-19T03:13:18.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:18 smithi059 ceph-mon[29133]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-19T03:13:18.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:18 smithi059 ceph-mon[29133]: osdmap e43: 6 total, 6 up, 6 in 2024-03-19T03:13:18.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:18 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-19T03:13:18.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:18 smithi059 ceph-mon[29133]: fsmap cephfs:0 2024-03-19T03:13:18.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:18 smithi059 ceph-mon[29133]: Saving service mds.cephfs spec with placement count:4 2024-03-19T03:13:18.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:18 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:18.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:18 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:13:18.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:18 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:13:18.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:18 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:13:18.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:18 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:18.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:18 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.hmylwa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:13:18.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:18 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.hmylwa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:13:18.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:18 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:13:18.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:18 smithi059 ceph-mon[29133]: Deploying daemon mds.cephfs.smithi059.hmylwa on smithi059 2024-03-19T03:13:18.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:18 smithi176 ceph-mon[36450]: pgmap v172: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:18.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:18 smithi176 ceph-mon[36450]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T03:13:18.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:18 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-19T03:13:18.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:18 smithi176 ceph-mon[36450]: osdmap e42: 6 total, 6 up, 6 in 2024-03-19T03:13:18.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:18 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-19T03:13:18.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:18 smithi176 ceph-mon[36450]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-19T03:13:18.380 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:18 smithi176 ceph-mon[36450]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-19T03:13:18.380 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:18 smithi176 ceph-mon[36450]: osdmap e43: 6 total, 6 up, 6 in 2024-03-19T03:13:18.380 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:18 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-19T03:13:18.380 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:18 smithi176 ceph-mon[36450]: fsmap cephfs:0 2024-03-19T03:13:18.380 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:18 smithi176 ceph-mon[36450]: Saving service mds.cephfs spec with placement count:4 2024-03-19T03:13:18.380 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:18 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:18.380 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:18 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:13:18.381 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:18 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:13:18.381 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:18 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:13:18.381 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:18 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:18.381 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:18 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.hmylwa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:13:18.381 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:18 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.hmylwa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:13:18.381 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:18 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:13:18.382 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:18 smithi176 ceph-mon[36450]: Deploying daemon mds.cephfs.smithi059.hmylwa on smithi059 2024-03-19T03:13:19.088 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:13:19.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:19 smithi059 ceph-mon[29133]: osdmap e44: 6 total, 6 up, 6 in 2024-03-19T03:13:19.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:19 smithi176 ceph-mon[36450]: osdmap e44: 6 total, 6 up, 6 in 2024-03-19T03:13:20.318 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:20 smithi059 ceph-mon[29133]: pgmap v176: 65 pgs: 26 active+clean, 3 creating+activating, 9 creating+peering, 27 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:20.318 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:20 smithi059 ceph-mon[29133]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T03:13:20.318 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:20 smithi059 ceph-mon[29133]: osdmap e45: 6 total, 6 up, 6 in 2024-03-19T03:13:20.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:20 smithi176 ceph-mon[36450]: pgmap v176: 65 pgs: 26 active+clean, 3 creating+activating, 9 creating+peering, 27 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:20.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:20 smithi176 ceph-mon[36450]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T03:13:20.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:20 smithi176 ceph-mon[36450]: osdmap e45: 6 total, 6 up, 6 in 2024-03-19T03:13:21.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.266+0000 7feef12a1700 1 -- 172.21.15.59:0/2869802182 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feeec0fbcd0 msgr2=0x7feeec0fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:21.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.266+0000 7feef12a1700 1 --2- 172.21.15.59:0/2869802182 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feeec0fbcd0 0x7feeec0fc0e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7feedc004940 tx=0x7feedc009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:21.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.266+0000 7feef12a1700 1 -- 172.21.15.59:0/2869802182 shutdown_connections 2024-03-19T03:13:21.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.266+0000 7feef12a1700 1 --2- 172.21.15.59:0/2869802182 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feeec0fcf80 0x7feeec0fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:21.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.266+0000 7feef12a1700 1 --2- 172.21.15.59:0/2869802182 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feeec0fbcd0 0x7feeec0fc0e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:21.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.266+0000 7feef12a1700 1 -- 172.21.15.59:0/2869802182 >> 172.21.15.59:0/2869802182 conn(0x7feeec0f7430 msgr2=0x7feeec0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:21.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.266+0000 7feef12a1700 1 -- 172.21.15.59:0/2869802182 shutdown_connections 2024-03-19T03:13:21.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.266+0000 7feef12a1700 1 -- 172.21.15.59:0/2869802182 wait complete. 2024-03-19T03:13:21.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.267+0000 7feef12a1700 1 Processor -- start 2024-03-19T03:13:21.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.267+0000 7feef12a1700 1 -- start start 2024-03-19T03:13:21.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.267+0000 7feef12a1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feeec0fbcd0 0x7feeec110720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:21.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.267+0000 7feef12a1700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feeec0fcf80 0x7feeec112c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:21.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.267+0000 7feef12a1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeec066350 con 0x7feeec0fbcd0 2024-03-19T03:13:21.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.267+0000 7feef12a1700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeec066490 con 0x7feeec0fcf80 2024-03-19T03:13:21.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.267+0000 7feeeaffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feeec0fbcd0 0x7feeec110720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:21.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.267+0000 7feeeaffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feeec0fbcd0 0x7feeec110720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51458/0 (socket says 172.21.15.59:51458) 2024-03-19T03:13:21.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.267+0000 7feeeaffd700 1 -- 172.21.15.59:0/2264437419 learned_addr learned my addr 172.21.15.59:0/2264437419 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:13:21.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.267+0000 7feeea7fc700 1 --2- 172.21.15.59:0/2264437419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feeec0fcf80 0x7feeec112c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:21.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.267+0000 7feeeaffd700 1 -- 172.21.15.59:0/2264437419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feeec0fcf80 msgr2=0x7feeec112c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:21.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.267+0000 7feeeaffd700 1 --2- 172.21.15.59:0/2264437419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feeec0fcf80 0x7feeec112c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:21.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.267+0000 7feeeaffd700 1 -- 172.21.15.59:0/2264437419 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feedc005040 con 0x7feeec0fbcd0 2024-03-19T03:13:21.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.267+0000 7feeea7fc700 1 --2- 172.21.15.59:0/2264437419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feeec0fcf80 0x7feeec112c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:13:21.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.267+0000 7feeeaffd700 1 --2- 172.21.15.59:0/2264437419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feeec0fbcd0 0x7feeec110720 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7feedc009150 tx=0x7feedc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:21.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.267+0000 7feed37fe700 1 -- 172.21.15.59:0/2264437419 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feedc00e040 con 0x7feeec0fbcd0 2024-03-19T03:13:21.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.267+0000 7feef12a1700 1 -- 172.21.15.59:0/2264437419 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeec1131b0 con 0x7feeec0fbcd0 2024-03-19T03:13:21.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.268+0000 7feed37fe700 1 -- 172.21.15.59:0/2264437419 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feedc00bcc0 con 0x7feeec0fbcd0 2024-03-19T03:13:21.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.268+0000 7feef12a1700 1 -- 172.21.15.59:0/2264437419 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeec113770 con 0x7feeec0fbcd0 2024-03-19T03:13:21.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.268+0000 7feed37fe700 1 -- 172.21.15.59:0/2264437419 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feedc01c630 con 0x7feeec0fbcd0 2024-03-19T03:13:21.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.268+0000 7feef12a1700 1 -- 172.21.15.59:0/2264437419 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeec05f5f0 con 0x7feeec0fbcd0 2024-03-19T03:13:21.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.269+0000 7feed37fe700 1 -- 172.21.15.59:0/2264437419 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7feedc013420 con 0x7feeec0fbcd0 2024-03-19T03:13:21.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.269+0000 7feed37fe700 1 --2- 172.21.15.59:0/2264437419 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7feed40692a0 0x7feed406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:21.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.269+0000 7feed37fe700 1 -- 172.21.15.59:0/2264437419 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7feedc087fd0 con 0x7feeec0fbcd0 2024-03-19T03:13:21.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.269+0000 7feeea7fc700 1 --2- 172.21.15.59:0/2264437419 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7feed40692a0 0x7feed406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:21.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.270+0000 7feeea7fc700 1 --2- 172.21.15.59:0/2264437419 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7feed40692a0 0x7feed406b750 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7feee0009fd0 tx=0x7feee0009ee0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:21.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.274+0000 7feed37fe700 1 -- 172.21.15.59:0/2264437419 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7feedc054300 con 0x7feeec0fbcd0 2024-03-19T03:13:21.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.520+0000 7feef12a1700 1 -- 172.21.15.59:0/2264437419 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7feeec113aa0 con 0x7feeec0fbcd0 2024-03-19T03:13:21.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.521+0000 7feed37fe700 1 -- 172.21.15.59:0/2264437419 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 4 v4) v1 ==== 75+0+1247 (secure 0 0 0) 0x7feedc057920 con 0x7feeec0fbcd0 2024-03-19T03:13:21.523 INFO:teuthology.orchestra.run.smithi059.stdout:e4 2024-03-19T03:13:21.523 INFO:teuthology.orchestra.run.smithi059.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-19T03:13:21.523 INFO:teuthology.orchestra.run.smithi059.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-19T03:13:21.523 INFO:teuthology.orchestra.run.smithi059.stdout:legacy client fscid: 1 2024-03-19T03:13:21.523 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:13:21.523 INFO:teuthology.orchestra.run.smithi059.stdout:Filesystem 'cephfs' (1) 2024-03-19T03:13:21.523 INFO:teuthology.orchestra.run.smithi059.stdout:fs_name cephfs 2024-03-19T03:13:21.524 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 4 2024-03-19T03:13:21.524 INFO:teuthology.orchestra.run.smithi059.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-19T03:13:21.524 INFO:teuthology.orchestra.run.smithi059.stdout:created 2024-03-19T03:13:17.039133+0000 2024-03-19T03:13:21.524 INFO:teuthology.orchestra.run.smithi059.stdout:modified 2024-03-19T03:13:21.401613+0000 2024-03-19T03:13:21.524 INFO:teuthology.orchestra.run.smithi059.stdout:tableserver 0 2024-03-19T03:13:21.524 INFO:teuthology.orchestra.run.smithi059.stdout:root 0 2024-03-19T03:13:21.524 INFO:teuthology.orchestra.run.smithi059.stdout:session_timeout 60 2024-03-19T03:13:21.524 INFO:teuthology.orchestra.run.smithi059.stdout:session_autoclose 300 2024-03-19T03:13:21.524 INFO:teuthology.orchestra.run.smithi059.stdout:max_file_size 1099511627776 2024-03-19T03:13:21.524 INFO:teuthology.orchestra.run.smithi059.stdout:required_client_features {} 2024-03-19T03:13:21.524 INFO:teuthology.orchestra.run.smithi059.stdout:last_failure 0 2024-03-19T03:13:21.524 INFO:teuthology.orchestra.run.smithi059.stdout:last_failure_osd_epoch 0 2024-03-19T03:13:21.524 INFO:teuthology.orchestra.run.smithi059.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-19T03:13:21.524 INFO:teuthology.orchestra.run.smithi059.stdout:max_mds 1 2024-03-19T03:13:21.524 INFO:teuthology.orchestra.run.smithi059.stdout:in 0 2024-03-19T03:13:21.525 INFO:teuthology.orchestra.run.smithi059.stdout:up {0=14468} 2024-03-19T03:13:21.525 INFO:teuthology.orchestra.run.smithi059.stdout:failed 2024-03-19T03:13:21.525 INFO:teuthology.orchestra.run.smithi059.stdout:damaged 2024-03-19T03:13:21.525 INFO:teuthology.orchestra.run.smithi059.stdout:stopped 2024-03-19T03:13:21.525 INFO:teuthology.orchestra.run.smithi059.stdout:data_pools [3] 2024-03-19T03:13:21.525 INFO:teuthology.orchestra.run.smithi059.stdout:metadata_pool 2 2024-03-19T03:13:21.525 INFO:teuthology.orchestra.run.smithi059.stdout:inline_data disabled 2024-03-19T03:13:21.525 INFO:teuthology.orchestra.run.smithi059.stdout:balancer 2024-03-19T03:13:21.525 INFO:teuthology.orchestra.run.smithi059.stdout:standby_count_wanted 0 2024-03-19T03:13:21.525 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi059.hmylwa{0:14468} state up:creating seq 1 addr [v2:172.21.15.59:6826/2936511711,v1:172.21.15.59:6827/2936511711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-19T03:13:21.525 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:13:21.525 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:13:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.523+0000 7feef12a1700 1 -- 172.21.15.59:0/2264437419 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7feed40692a0 msgr2=0x7feed406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.523+0000 7feef12a1700 1 --2- 172.21.15.59:0/2264437419 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7feed40692a0 0x7feed406b750 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7feee0009fd0 tx=0x7feee0009ee0 comp rx=0 tx=0).stop 2024-03-19T03:13:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.523+0000 7feef12a1700 1 -- 172.21.15.59:0/2264437419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feeec0fbcd0 msgr2=0x7feeec110720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.523+0000 7feef12a1700 1 --2- 172.21.15.59:0/2264437419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feeec0fbcd0 0x7feeec110720 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7feedc009150 tx=0x7feedc009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.523+0000 7feef12a1700 1 -- 172.21.15.59:0/2264437419 shutdown_connections 2024-03-19T03:13:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.523+0000 7feef12a1700 1 --2- 172.21.15.59:0/2264437419 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7feed40692a0 0x7feed406b750 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.523+0000 7feef12a1700 1 --2- 172.21.15.59:0/2264437419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feeec0fbcd0 0x7feeec110720 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.523+0000 7feef12a1700 1 --2- 172.21.15.59:0/2264437419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feeec0fcf80 0x7feeec112c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.523+0000 7feef12a1700 1 -- 172.21.15.59:0/2264437419 >> 172.21.15.59:0/2264437419 conn(0x7feeec0f7430 msgr2=0x7feeec0f8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.523+0000 7feef12a1700 1 -- 172.21.15.59:0/2264437419 shutdown_connections 2024-03-19T03:13:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:21.523+0000 7feef12a1700 1 -- 172.21.15.59:0/2264437419 wait complete. 2024-03-19T03:13:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 4 2024-03-19T03:13:22.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:22 smithi176 ceph-mon[36450]: pgmap v178: 65 pgs: 47 active+clean, 3 creating+activating, 9 creating+peering, 6 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:22.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:22 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:22.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:22 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:22.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:22 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:22.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:22 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi176.bochhq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:13:22.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:22 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi176.bochhq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:13:22.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:22 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:13:22.130 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:22 smithi176 ceph-mon[36450]: Deploying daemon mds.cephfs.smithi176.bochhq on smithi176 2024-03-19T03:13:22.130 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:22 smithi176 ceph-mon[36450]: mds.? [v2:172.21.15.59:6826/2936511711,v1:172.21.15.59:6827/2936511711] up:boot 2024-03-19T03:13:22.130 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:22 smithi176 ceph-mon[36450]: daemon mds.cephfs.smithi059.hmylwa assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-19T03:13:22.130 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:22 smithi176 ceph-mon[36450]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-19T03:13:22.130 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:22 smithi176 ceph-mon[36450]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-19T03:13:22.130 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:22 smithi176 ceph-mon[36450]: Cluster is now healthy 2024-03-19T03:13:22.130 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:22 smithi176 ceph-mon[36450]: fsmap cephfs:0 1 up:standby 2024-03-19T03:13:22.131 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:22 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi059.hmylwa"}]: dispatch 2024-03-19T03:13:22.131 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:22 smithi176 ceph-mon[36450]: fsmap cephfs:1 {0=cephfs.smithi059.hmylwa=up:creating} 2024-03-19T03:13:22.131 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:22 smithi176 ceph-mon[36450]: daemon mds.cephfs.smithi059.hmylwa is now active in filesystem cephfs as rank 0 2024-03-19T03:13:22.131 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:22 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/2264437419' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-19T03:13:22.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:22 smithi059 ceph-mon[29133]: pgmap v178: 65 pgs: 47 active+clean, 3 creating+activating, 9 creating+peering, 6 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:22.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:22 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:22.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:22 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:22.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:22 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:22.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:22 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi176.bochhq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:13:22.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:22 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi176.bochhq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:13:22.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:22 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:13:22.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:22 smithi059 ceph-mon[29133]: Deploying daemon mds.cephfs.smithi176.bochhq on smithi176 2024-03-19T03:13:22.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:22 smithi059 ceph-mon[29133]: mds.? [v2:172.21.15.59:6826/2936511711,v1:172.21.15.59:6827/2936511711] up:boot 2024-03-19T03:13:22.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:22 smithi059 ceph-mon[29133]: daemon mds.cephfs.smithi059.hmylwa assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-19T03:13:22.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:22 smithi059 ceph-mon[29133]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-19T03:13:22.142 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:22 smithi059 ceph-mon[29133]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-19T03:13:22.142 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:22 smithi059 ceph-mon[29133]: Cluster is now healthy 2024-03-19T03:13:22.142 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:22 smithi059 ceph-mon[29133]: fsmap cephfs:0 1 up:standby 2024-03-19T03:13:22.142 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:22 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi059.hmylwa"}]: dispatch 2024-03-19T03:13:22.143 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:22 smithi059 ceph-mon[29133]: fsmap cephfs:1 {0=cephfs.smithi059.hmylwa=up:creating} 2024-03-19T03:13:22.143 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:22 smithi059 ceph-mon[29133]: daemon mds.cephfs.smithi059.hmylwa is now active in filesystem cephfs as rank 0 2024-03-19T03:13:22.143 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:22 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/2264437419' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-19T03:13:22.162 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-19T03:13:22.570 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2024-03-19T03:13:22.570 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 1' 2024-03-19T03:13:22.845 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:13:23.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:23 smithi059 ceph-mon[29133]: mds.? [v2:172.21.15.59:6826/2936511711,v1:172.21.15.59:6827/2936511711] up:active 2024-03-19T03:13:23.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:23 smithi059 ceph-mon[29133]: fsmap cephfs:1 {0=cephfs.smithi059.hmylwa=up:active} 2024-03-19T03:13:23.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:23 smithi059 ceph-mon[29133]: pgmap v179: 65 pgs: 62 active+clean, 3 creating+activating; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 3 op/s 2024-03-19T03:13:23.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:23 smithi176 ceph-mon[36450]: mds.? [v2:172.21.15.59:6826/2936511711,v1:172.21.15.59:6827/2936511711] up:active 2024-03-19T03:13:23.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:23 smithi176 ceph-mon[36450]: fsmap cephfs:1 {0=cephfs.smithi059.hmylwa=up:active} 2024-03-19T03:13:23.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:23 smithi176 ceph-mon[36450]: pgmap v179: 65 pgs: 62 active+clean, 3 creating+activating; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 3 op/s 2024-03-19T03:13:24.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.465+0000 7fc9cd3c1700 1 -- 172.21.15.59:0/3249137697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c80fdfe0 msgr2=0x7fc9c80fe450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.465+0000 7fc9cd3c1700 1 --2- 172.21.15.59:0/3249137697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c80fdfe0 0x7fc9c80fe450 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc004940 tx=0x7fc9bc009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.465+0000 7fc9cd3c1700 1 -- 172.21.15.59:0/3249137697 shutdown_connections 2024-03-19T03:13:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.465+0000 7fc9cd3c1700 1 --2- 172.21.15.59:0/3249137697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c80fdfe0 0x7fc9c80fe450 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.465+0000 7fc9cd3c1700 1 --2- 172.21.15.59:0/3249137697 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9c80fce70 0x7fc9c80fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.465+0000 7fc9cd3c1700 1 -- 172.21.15.59:0/3249137697 >> 172.21.15.59:0/3249137697 conn(0x7fc9c80f8550 msgr2=0x7fc9c80fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.465+0000 7fc9cd3c1700 1 -- 172.21.15.59:0/3249137697 shutdown_connections 2024-03-19T03:13:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.465+0000 7fc9cd3c1700 1 -- 172.21.15.59:0/3249137697 wait complete. 2024-03-19T03:13:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.466+0000 7fc9cd3c1700 1 Processor -- start 2024-03-19T03:13:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.466+0000 7fc9cd3c1700 1 -- start start 2024-03-19T03:13:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.466+0000 7fc9cd3c1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c80fce70 0x7fc9c8071c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.466+0000 7fc9cd3c1700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9c80fdfe0 0x7fc9c8070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.466+0000 7fc9cd3c1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c8070850 con 0x7fc9c80fce70 2024-03-19T03:13:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.466+0000 7fc9cd3c1700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c80709c0 con 0x7fc9c80fdfe0 2024-03-19T03:13:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.466+0000 7fc9c6ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c80fce70 0x7fc9c8071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.466+0000 7fc9c6ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c80fce70 0x7fc9c8071c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47996/0 (socket says 172.21.15.59:47996) 2024-03-19T03:13:24.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.466+0000 7fc9c6ffd700 1 -- 172.21.15.59:0/2953416999 learned_addr learned my addr 172.21.15.59:0/2953416999 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:13:24.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.466+0000 7fc9c6ffd700 1 -- 172.21.15.59:0/2953416999 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9c80fdfe0 msgr2=0x7fc9c8070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:24.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.466+0000 7fc9c67fc700 1 --2- 172.21.15.59:0/2953416999 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9c80fdfe0 0x7fc9c8070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:24.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.466+0000 7fc9c6ffd700 1 --2- 172.21.15.59:0/2953416999 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9c80fdfe0 0x7fc9c8070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:24.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.466+0000 7fc9c6ffd700 1 -- 172.21.15.59:0/2953416999 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9bc005040 con 0x7fc9c80fce70 2024-03-19T03:13:24.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.466+0000 7fc9c6ffd700 1 --2- 172.21.15.59:0/2953416999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c80fce70 0x7fc9c8071c60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc9b800b790 tx=0x7fc9b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:24.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.466+0000 7fc9af7fe700 1 -- 172.21.15.59:0/2953416999 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9b800f770 con 0x7fc9c80fce70 2024-03-19T03:13:24.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.467+0000 7fc9af7fe700 1 -- 172.21.15.59:0/2953416999 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc9b800fda0 con 0x7fc9c80fce70 2024-03-19T03:13:24.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.467+0000 7fc9cd3c1700 1 -- 172.21.15.59:0/2953416999 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c8070ca0 con 0x7fc9c80fce70 2024-03-19T03:13:24.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.467+0000 7fc9af7fe700 1 -- 172.21.15.59:0/2953416999 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9b800d310 con 0x7fc9c80fce70 2024-03-19T03:13:24.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.467+0000 7fc9cd3c1700 1 -- 172.21.15.59:0/2953416999 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c8115a40 con 0x7fc9c80fce70 2024-03-19T03:13:24.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.468+0000 7fc9cd3c1700 1 -- 172.21.15.59:0/2953416999 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c804c710 con 0x7fc9c80fce70 2024-03-19T03:13:24.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.468+0000 7fc9af7fe700 1 -- 172.21.15.59:0/2953416999 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc9b8010420 con 0x7fc9c80fce70 2024-03-19T03:13:24.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.468+0000 7fc9af7fe700 1 --2- 172.21.15.59:0/2953416999 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc9b00692a0 0x7fc9b006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:24.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.469+0000 7fc9af7fe700 1 -- 172.21.15.59:0/2953416999 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fc9b80871a0 con 0x7fc9c80fce70 2024-03-19T03:13:24.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.469+0000 7fc9c67fc700 1 --2- 172.21.15.59:0/2953416999 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc9b00692a0 0x7fc9b006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:24.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.469+0000 7fc9c67fc700 1 --2- 172.21.15.59:0/2953416999 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc9b00692a0 0x7fc9b006b750 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc000c00 tx=0x7fc9bc009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:24.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.473+0000 7fc9af7fe700 1 -- 172.21.15.59:0/2953416999 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fc9b8056f60 con 0x7fc9c80fce70 2024-03-19T03:13:24.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:24 smithi059 ceph-mon[29133]: mds.? [v2:172.21.15.176:6824/193753714,v1:172.21.15.176:6825/193753714] up:boot 2024-03-19T03:13:24.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:24 smithi059 ceph-mon[29133]: fsmap cephfs:1 {0=cephfs.smithi059.hmylwa=up:active} 1 up:standby 2024-03-19T03:13:24.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:24 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi176.bochhq"}]: dispatch 2024-03-19T03:13:24.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:24 smithi059 ceph-mon[29133]: fsmap cephfs:1 {0=cephfs.smithi059.hmylwa=up:active} 1 up:standby 2024-03-19T03:13:24.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:24 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:24.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:24 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:24.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:24 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:24.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:24 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.doeuxg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:13:24.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:24 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.doeuxg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:13:24.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:24 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:13:24.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:24 smithi059 ceph-mon[29133]: Deploying daemon mds.cephfs.smithi059.doeuxg on smithi059 2024-03-19T03:13:24.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:24.723+0000 7fc9cd3c1700 1 -- 172.21.15.59:0/2953416999 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7fc9c8115bd0 con 0x7fc9c80fce70 2024-03-19T03:13:24.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:24 smithi176 ceph-mon[36450]: mds.? [v2:172.21.15.176:6824/193753714,v1:172.21.15.176:6825/193753714] up:boot 2024-03-19T03:13:24.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:24 smithi176 ceph-mon[36450]: fsmap cephfs:1 {0=cephfs.smithi059.hmylwa=up:active} 1 up:standby 2024-03-19T03:13:24.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:24 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi176.bochhq"}]: dispatch 2024-03-19T03:13:24.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:24 smithi176 ceph-mon[36450]: fsmap cephfs:1 {0=cephfs.smithi059.hmylwa=up:active} 1 up:standby 2024-03-19T03:13:24.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:24 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:24.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:24 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:24.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:24 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:24.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:24 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.doeuxg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:13:24.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:24 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.doeuxg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:13:24.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:24 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:13:24.879 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:24 smithi176 ceph-mon[36450]: Deploying daemon mds.cephfs.smithi059.doeuxg on smithi059 2024-03-19T03:13:25.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:25.419+0000 7fc9af7fe700 1 -- 172.21.15.59:0/2953416999 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v8) v1 ==== 105+0+0 (secure 0 0 0) 0x7fc9b8056af0 con 0x7fc9c80fce70 2024-03-19T03:13:25.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:25.421+0000 7fc9cd3c1700 1 -- 172.21.15.59:0/2953416999 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc9b00692a0 msgr2=0x7fc9b006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:25.421+0000 7fc9cd3c1700 1 --2- 172.21.15.59:0/2953416999 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc9b00692a0 0x7fc9b006b750 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc000c00 tx=0x7fc9bc009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:25.421+0000 7fc9cd3c1700 1 -- 172.21.15.59:0/2953416999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c80fce70 msgr2=0x7fc9c8071c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:25.421+0000 7fc9cd3c1700 1 --2- 172.21.15.59:0/2953416999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c80fce70 0x7fc9c8071c60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc9b800b790 tx=0x7fc9b8009250 comp rx=0 tx=0).stop 2024-03-19T03:13:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:25.421+0000 7fc9cd3c1700 1 -- 172.21.15.59:0/2953416999 shutdown_connections 2024-03-19T03:13:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:25.421+0000 7fc9cd3c1700 1 --2- 172.21.15.59:0/2953416999 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc9b00692a0 0x7fc9b006b750 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:25.421+0000 7fc9cd3c1700 1 --2- 172.21.15.59:0/2953416999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c80fce70 0x7fc9c8071c60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:25.421+0000 7fc9cd3c1700 1 --2- 172.21.15.59:0/2953416999 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9c80fdfe0 0x7fc9c8070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:25.421+0000 7fc9cd3c1700 1 -- 172.21.15.59:0/2953416999 >> 172.21.15.59:0/2953416999 conn(0x7fc9c80f8550 msgr2=0x7fc9c80f9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:25.421+0000 7fc9cd3c1700 1 -- 172.21.15.59:0/2953416999 shutdown_connections 2024-03-19T03:13:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:25.421+0000 7fc9cd3c1700 1 -- 172.21.15.59:0/2953416999 wait complete. 2024-03-19T03:13:25.528 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:25 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/2953416999' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-03-19T03:13:25.528 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:25 smithi059 ceph-mon[29133]: pgmap v180: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-03-19T03:13:25.818 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:25 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/2953416999' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-03-19T03:13:25.818 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:25 smithi176 ceph-mon[36450]: pgmap v180: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-03-19T03:13:26.142 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-19T03:13:26.152 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2024-03-19T03:13:26.152 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-03-19T03:13:26.459 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:26 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/2953416999' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-03-19T03:13:26.459 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:26 smithi059 ceph-mon[29133]: fsmap cephfs:1 {0=cephfs.smithi059.hmylwa=up:active} 1 up:standby 2024-03-19T03:13:26.793 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:13:26.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:26 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/2953416999' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-03-19T03:13:26.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:26 smithi176 ceph-mon[36450]: fsmap cephfs:1 {0=cephfs.smithi059.hmylwa=up:active} 1 up:standby 2024-03-19T03:13:27.926 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:27 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:27.926 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:27 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:27.926 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:27 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:27.926 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:27 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi176.gisnqc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:13:27.926 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:27 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi176.gisnqc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:13:27.926 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:27 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:13:27.926 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:27 smithi059 ceph-mon[29133]: Deploying daemon mds.cephfs.smithi176.gisnqc on smithi176 2024-03-19T03:13:27.926 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:27 smithi059 ceph-mon[29133]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s wr, 4 op/s 2024-03-19T03:13:27.927 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:27 smithi059 ceph-mon[29133]: mds.? [v2:172.21.15.59:6828/503726080,v1:172.21.15.59:6829/503726080] up:boot 2024-03-19T03:13:27.927 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:27 smithi059 ceph-mon[29133]: fsmap cephfs:1 {0=cephfs.smithi059.hmylwa=up:active} 2 up:standby 2024-03-19T03:13:27.927 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:27 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi059.doeuxg"}]: dispatch 2024-03-19T03:13:28.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:27 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:28.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:27 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:28.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:27 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:28.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:27 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi176.gisnqc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:13:28.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:27 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi176.gisnqc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:13:28.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:27 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:13:28.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:27 smithi176 ceph-mon[36450]: Deploying daemon mds.cephfs.smithi176.gisnqc on smithi176 2024-03-19T03:13:28.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:27 smithi176 ceph-mon[36450]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s wr, 4 op/s 2024-03-19T03:13:28.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:27 smithi176 ceph-mon[36450]: mds.? [v2:172.21.15.59:6828/503726080,v1:172.21.15.59:6829/503726080] up:boot 2024-03-19T03:13:28.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:27 smithi176 ceph-mon[36450]: fsmap cephfs:1 {0=cephfs.smithi059.hmylwa=up:active} 2 up:standby 2024-03-19T03:13:28.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:27 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi059.doeuxg"}]: dispatch 2024-03-19T03:13:28.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.448+0000 7fb7da74c700 1 -- 172.21.15.59:0/4149314863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7d40fce70 msgr2=0x7fb7d40fd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:28.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.448+0000 7fb7da74c700 1 --2- 172.21.15.59:0/4149314863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7d40fce70 0x7fb7d40fd280 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4004940 tx=0x7fb7c4009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:28.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.448+0000 7fb7da74c700 1 -- 172.21.15.59:0/4149314863 shutdown_connections 2024-03-19T03:13:28.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.448+0000 7fb7da74c700 1 --2- 172.21.15.59:0/4149314863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7d40fdfe0 0x7fb7d40fe450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:28.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.448+0000 7fb7da74c700 1 --2- 172.21.15.59:0/4149314863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7d40fce70 0x7fb7d40fd280 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:28.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.448+0000 7fb7da74c700 1 -- 172.21.15.59:0/4149314863 >> 172.21.15.59:0/4149314863 conn(0x7fb7d40f8550 msgr2=0x7fb7d40fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:28.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.448+0000 7fb7da74c700 1 -- 172.21.15.59:0/4149314863 shutdown_connections 2024-03-19T03:13:28.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.448+0000 7fb7da74c700 1 -- 172.21.15.59:0/4149314863 wait complete. 2024-03-19T03:13:28.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.448+0000 7fb7da74c700 1 Processor -- start 2024-03-19T03:13:28.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.448+0000 7fb7da74c700 1 -- start start 2024-03-19T03:13:28.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.448+0000 7fb7da74c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7d40fce70 0x7fb7d4071c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.448+0000 7fb7da74c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7d40fdfe0 0x7fb7d4070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.448+0000 7fb7da74c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d4072260 con 0x7fb7d40fce70 2024-03-19T03:13:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.448+0000 7fb7da74c700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d4070850 con 0x7fb7d40fdfe0 2024-03-19T03:13:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.449+0000 7fb7d3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7d40fce70 0x7fb7d4071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.449+0000 7fb7d3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7d40fce70 0x7fb7d4071c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48038/0 (socket says 172.21.15.59:48038) 2024-03-19T03:13:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.449+0000 7fb7d3fff700 1 -- 172.21.15.59:0/2695139405 learned_addr learned my addr 172.21.15.59:0/2695139405 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:13:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.449+0000 7fb7d37fe700 1 --2- 172.21.15.59:0/2695139405 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7d40fdfe0 0x7fb7d4070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.449+0000 7fb7d3fff700 1 -- 172.21.15.59:0/2695139405 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7d40fdfe0 msgr2=0x7fb7d4070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.449+0000 7fb7d3fff700 1 --2- 172.21.15.59:0/2695139405 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7d40fdfe0 0x7fb7d4070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.449+0000 7fb7d3fff700 1 -- 172.21.15.59:0/2695139405 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c4005040 con 0x7fb7d40fce70 2024-03-19T03:13:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.449+0000 7fb7d3fff700 1 --2- 172.21.15.59:0/2695139405 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7d40fce70 0x7fb7d4071c60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4013870 tx=0x7fb7c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.449+0000 7fb7d0ff9700 1 -- 172.21.15.59:0/2695139405 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7c4021070 con 0x7fb7d40fce70 2024-03-19T03:13:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.449+0000 7fb7d0ff9700 1 -- 172.21.15.59:0/2695139405 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7c4009430 con 0x7fb7d40fce70 2024-03-19T03:13:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.449+0000 7fb7da74c700 1 -- 172.21.15.59:0/2695139405 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d4070aa0 con 0x7fb7d40fce70 2024-03-19T03:13:28.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.449+0000 7fb7d0ff9700 1 -- 172.21.15.59:0/2695139405 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7c400e070 con 0x7fb7d40fce70 2024-03-19T03:13:28.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.449+0000 7fb7da74c700 1 -- 172.21.15.59:0/2695139405 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d4070fe0 con 0x7fb7d40fce70 2024-03-19T03:13:28.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.450+0000 7fb7da74c700 1 -- 172.21.15.59:0/2695139405 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d404c710 con 0x7fb7d40fce70 2024-03-19T03:13:28.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.451+0000 7fb7d0ff9700 1 -- 172.21.15.59:0/2695139405 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb7c401c750 con 0x7fb7d40fce70 2024-03-19T03:13:28.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.451+0000 7fb7d0ff9700 1 --2- 172.21.15.59:0/2695139405 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb7c00692a0 0x7fb7c006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:28.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.451+0000 7fb7d0ff9700 1 -- 172.21.15.59:0/2695139405 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fb7c4088390 con 0x7fb7d40fce70 2024-03-19T03:13:28.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.451+0000 7fb7d37fe700 1 --2- 172.21.15.59:0/2695139405 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb7c00692a0 0x7fb7c006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:28.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.452+0000 7fb7d37fe700 1 --2- 172.21.15.59:0/2695139405 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb7c00692a0 0x7fb7c006b750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc009b80 tx=0x7fb7bc00b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:28.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.456+0000 7fb7d0ff9700 1 -- 172.21.15.59:0/2695139405 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fb7c40581f0 con 0x7fb7d40fce70 2024-03-19T03:13:28.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.692+0000 7fb7da74c700 1 -- 172.21.15.59:0/2695139405 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb7d41058d0 con 0x7fb7d40fce70 2024-03-19T03:13:28.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.783+0000 7fb7d0ff9700 1 -- 172.21.15.59:0/2695139405 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb7c4057d80 con 0x7fb7d40fce70 2024-03-19T03:13:28.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.785+0000 7fb7da74c700 1 -- 172.21.15.59:0/2695139405 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb7c00692a0 msgr2=0x7fb7c006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:28.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.785+0000 7fb7da74c700 1 --2- 172.21.15.59:0/2695139405 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb7c00692a0 0x7fb7c006b750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc009b80 tx=0x7fb7bc00b040 comp rx=0 tx=0).stop 2024-03-19T03:13:28.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.785+0000 7fb7da74c700 1 -- 172.21.15.59:0/2695139405 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7d40fce70 msgr2=0x7fb7d4071c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:28.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.785+0000 7fb7da74c700 1 --2- 172.21.15.59:0/2695139405 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7d40fce70 0x7fb7d4071c60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4013870 tx=0x7fb7c4009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:28.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.785+0000 7fb7da74c700 1 -- 172.21.15.59:0/2695139405 shutdown_connections 2024-03-19T03:13:28.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.785+0000 7fb7da74c700 1 --2- 172.21.15.59:0/2695139405 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb7c00692a0 0x7fb7c006b750 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:28.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.785+0000 7fb7da74c700 1 --2- 172.21.15.59:0/2695139405 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7d40fce70 0x7fb7d4071c60 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:28.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.785+0000 7fb7da74c700 1 --2- 172.21.15.59:0/2695139405 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7d40fdfe0 0x7fb7d4070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:28.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.785+0000 7fb7da74c700 1 -- 172.21.15.59:0/2695139405 >> 172.21.15.59:0/2695139405 conn(0x7fb7d40f8550 msgr2=0x7fb7d40f9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:28.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.785+0000 7fb7da74c700 1 -- 172.21.15.59:0/2695139405 shutdown_connections 2024-03-19T03:13:28.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:28.785+0000 7fb7da74c700 1 -- 172.21.15.59:0/2695139405 wait complete. 2024-03-19T03:13:29.058 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:28 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/2695139405' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-03-19T03:13:29.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:28 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/2695139405' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-03-19T03:13:29.424 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-19T03:13:29.434 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2024-03-19T03:13:29.434 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-03-19T03:13:29.708 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:13:30.052 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:29 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/2695139405' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-03-19T03:13:30.052 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:29 smithi059 ceph-mon[29133]: fsmap cephfs:1 {0=cephfs.smithi059.hmylwa=up:active} 2 up:standby 2024-03-19T03:13:30.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:29 smithi059 ceph-mon[29133]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-19T03:13:30.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:29 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:30.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:29 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:30.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:29 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:30.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:29 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:30.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:29 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:30.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:29 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:13:30.112 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:29 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/2695139405' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-03-19T03:13:30.113 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:29 smithi176 ceph-mon[36450]: fsmap cephfs:1 {0=cephfs.smithi059.hmylwa=up:active} 2 up:standby 2024-03-19T03:13:30.113 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:29 smithi176 ceph-mon[36450]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-19T03:13:30.113 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:29 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:30.113 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:29 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:30.113 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:29 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:30.113 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:29 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:30.113 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:29 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:30.113 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:29 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:13:30.903 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:30 smithi176 ceph-mon[36450]: mds.? [v2:172.21.15.176:6826/4058958323,v1:172.21.15.176:6827/4058958323] up:boot 2024-03-19T03:13:30.903 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:30 smithi176 ceph-mon[36450]: mds.? [v2:172.21.15.59:6826/2936511711,v1:172.21.15.59:6827/2936511711] up:active 2024-03-19T03:13:30.903 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:30 smithi176 ceph-mon[36450]: fsmap cephfs:1 {0=cephfs.smithi059.hmylwa=up:active} 3 up:standby 2024-03-19T03:13:30.904 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:30 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi176.gisnqc"}]: dispatch 2024-03-19T03:13:31.054 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:30 smithi059 ceph-mon[29133]: mds.? [v2:172.21.15.176:6826/4058958323,v1:172.21.15.176:6827/4058958323] up:boot 2024-03-19T03:13:31.054 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:30 smithi059 ceph-mon[29133]: mds.? [v2:172.21.15.59:6826/2936511711,v1:172.21.15.59:6827/2936511711] up:active 2024-03-19T03:13:31.054 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:30 smithi059 ceph-mon[29133]: fsmap cephfs:1 {0=cephfs.smithi059.hmylwa=up:active} 3 up:standby 2024-03-19T03:13:31.054 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:30 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi176.gisnqc"}]: dispatch 2024-03-19T03:13:31.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.279+0000 7f3796eb5700 1 -- 172.21.15.59:0/1945370868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3790066260 msgr2=0x7f37900ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:31.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.279+0000 7f3796eb5700 1 --2- 172.21.15.59:0/1945370868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3790066260 0x7f37900ffa40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3780004940 tx=0x7f3780009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:31.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.279+0000 7f3796eb5700 1 -- 172.21.15.59:0/1945370868 shutdown_connections 2024-03-19T03:13:31.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.279+0000 7f3796eb5700 1 --2- 172.21.15.59:0/1945370868 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f37900fff80 0x7f3790102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:31.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.279+0000 7f3796eb5700 1 --2- 172.21.15.59:0/1945370868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3790066260 0x7f37900ffa40 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:31.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.279+0000 7f3796eb5700 1 -- 172.21.15.59:0/1945370868 >> 172.21.15.59:0/1945370868 conn(0x7f37900f7430 msgr2=0x7f37900f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:31.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.279+0000 7f3796eb5700 1 -- 172.21.15.59:0/1945370868 shutdown_connections 2024-03-19T03:13:31.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.279+0000 7f3796eb5700 1 -- 172.21.15.59:0/1945370868 wait complete. 2024-03-19T03:13:31.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.279+0000 7f3796eb5700 1 Processor -- start 2024-03-19T03:13:31.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.279+0000 7f3796eb5700 1 -- start start 2024-03-19T03:13:31.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.280+0000 7f3796eb5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3790066260 0x7f37900fd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:31.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.280+0000 7f3796eb5700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f37900fff80 0x7f37900fb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:31.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.280+0000 7f3796eb5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37900fbed0 con 0x7f3790066260 2024-03-19T03:13:31.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.280+0000 7f3796eb5700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37900fc010 con 0x7f37900fff80 2024-03-19T03:13:31.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.280+0000 7f3794c51700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3790066260 0x7f37900fd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:31.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.280+0000 7f3794c51700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3790066260 0x7f37900fd420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48054/0 (socket says 172.21.15.59:48054) 2024-03-19T03:13:31.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.280+0000 7f3794c51700 1 -- 172.21.15.59:0/2131937619 learned_addr learned my addr 172.21.15.59:0/2131937619 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:13:31.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.280+0000 7f378ffff700 1 --2- 172.21.15.59:0/2131937619 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f37900fff80 0x7f37900fb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:31.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.280+0000 7f3794c51700 1 -- 172.21.15.59:0/2131937619 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f37900fff80 msgr2=0x7f37900fb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:31.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.280+0000 7f3794c51700 1 --2- 172.21.15.59:0/2131937619 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f37900fff80 0x7f37900fb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:31.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.280+0000 7f3794c51700 1 -- 172.21.15.59:0/2131937619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3780005040 con 0x7f3790066260 2024-03-19T03:13:31.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.280+0000 7f3794c51700 1 --2- 172.21.15.59:0/2131937619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3790066260 0x7f37900fd420 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3780000c00 tx=0x7f3780009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:31.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.280+0000 7f378d7fa700 1 -- 172.21.15.59:0/2131937619 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f378000e030 con 0x7f3790066260 2024-03-19T03:13:31.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.280+0000 7f378d7fa700 1 -- 172.21.15.59:0/2131937619 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3780013de0 con 0x7f3790066260 2024-03-19T03:13:31.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.280+0000 7f378d7fa700 1 -- 172.21.15.59:0/2131937619 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f378001ce50 con 0x7f3790066260 2024-03-19T03:13:31.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.280+0000 7f3796eb5700 1 -- 172.21.15.59:0/2131937619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37900fc290 con 0x7f3790066260 2024-03-19T03:13:31.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.281+0000 7f3796eb5700 1 -- 172.21.15.59:0/2131937619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37900fc780 con 0x7f3790066260 2024-03-19T03:13:31.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.281+0000 7f3796eb5700 1 -- 172.21.15.59:0/2131937619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f379005f5f0 con 0x7f3790066260 2024-03-19T03:13:31.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.282+0000 7f378d7fa700 1 -- 172.21.15.59:0/2131937619 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f378001c400 con 0x7f3790066260 2024-03-19T03:13:31.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.282+0000 7f378d7fa700 1 --2- 172.21.15.59:0/2131937619 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f37780692a0 0x7f377806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:31.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.282+0000 7f378d7fa700 1 -- 172.21.15.59:0/2131937619 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f3780088b10 con 0x7f3790066260 2024-03-19T03:13:31.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.282+0000 7f378ffff700 1 --2- 172.21.15.59:0/2131937619 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f37780692a0 0x7f377806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:31.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.283+0000 7f378ffff700 1 --2- 172.21.15.59:0/2131937619 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f37780692a0 0x7f377806b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f37900fda00 tx=0x7f378400b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:31.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.287+0000 7f378d7fa700 1 -- 172.21.15.59:0/2131937619 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f3780054e40 con 0x7f3790066260 2024-03-19T03:13:31.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:31.534+0000 7f3796eb5700 1 -- 172.21.15.59:0/2131937619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3790115c60 con 0x7f3790066260 2024-03-19T03:13:32.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:31 smithi059 ceph-mon[29133]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-19T03:13:32.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:13:32.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:32.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:31 smithi059 ceph-mon[29133]: mds.? [v2:172.21.15.59:6828/503726080,v1:172.21.15.59:6829/503726080] up:standby 2024-03-19T03:13:32.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:31 smithi059 ceph-mon[29133]: mds.? [v2:172.21.15.176:6824/193753714,v1:172.21.15.176:6825/193753714] up:standby 2024-03-19T03:13:32.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:31 smithi059 ceph-mon[29133]: fsmap cephfs:1 {0=cephfs.smithi059.hmylwa=up:active} 3 up:standby 2024-03-19T03:13:32.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:32.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:32.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:31 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/2131937619' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:13:32.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:31 smithi176 ceph-mon[36450]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-19T03:13:32.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:31 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:13:32.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:31 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:32.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:31 smithi176 ceph-mon[36450]: mds.? [v2:172.21.15.59:6828/503726080,v1:172.21.15.59:6829/503726080] up:standby 2024-03-19T03:13:32.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:31 smithi176 ceph-mon[36450]: mds.? [v2:172.21.15.176:6824/193753714,v1:172.21.15.176:6825/193753714] up:standby 2024-03-19T03:13:32.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:31 smithi176 ceph-mon[36450]: fsmap cephfs:1 {0=cephfs.smithi059.hmylwa=up:active} 3 up:standby 2024-03-19T03:13:32.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:31 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:32.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:31 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:32.129 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:31 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/2131937619' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:13:32.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:32.436+0000 7f378d7fa700 1 -- 172.21.15.59:0/2131937619 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v13) v1 ==== 168+0+0 (secure 0 0 0) 0x7f3780058460 con 0x7f3790066260 2024-03-19T03:13:32.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:32.438+0000 7f3796eb5700 1 -- 172.21.15.59:0/2131937619 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f37780692a0 msgr2=0x7f377806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:32.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:32.438+0000 7f3796eb5700 1 --2- 172.21.15.59:0/2131937619 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f37780692a0 0x7f377806b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f37900fda00 tx=0x7f378400b040 comp rx=0 tx=0).stop 2024-03-19T03:13:32.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:32.438+0000 7f3796eb5700 1 -- 172.21.15.59:0/2131937619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3790066260 msgr2=0x7f37900fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:32.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:32.438+0000 7f3796eb5700 1 --2- 172.21.15.59:0/2131937619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3790066260 0x7f37900fd420 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3780000c00 tx=0x7f3780009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:32.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:32.439+0000 7f3796eb5700 1 -- 172.21.15.59:0/2131937619 shutdown_connections 2024-03-19T03:13:32.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:32.439+0000 7f3796eb5700 1 --2- 172.21.15.59:0/2131937619 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f37780692a0 0x7f377806b750 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:32.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:32.439+0000 7f3796eb5700 1 --2- 172.21.15.59:0/2131937619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3790066260 0x7f37900fd420 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:32.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:32.439+0000 7f3796eb5700 1 --2- 172.21.15.59:0/2131937619 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f37900fff80 0x7f37900fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:32.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:32.439+0000 7f3796eb5700 1 -- 172.21.15.59:0/2131937619 >> 172.21.15.59:0/2131937619 conn(0x7f37900f7430 msgr2=0x7f37900f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:32.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:32.439+0000 7f3796eb5700 1 -- 172.21.15.59:0/2131937619 shutdown_connections 2024-03-19T03:13:32.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:32.439+0000 7f3796eb5700 1 -- 172.21.15.59:0/2131937619 wait complete. 2024-03-19T03:13:32.441 INFO:teuthology.orchestra.run.smithi059.stderr:inline data enabled 2024-03-19T03:13:33.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:32 smithi059 ceph-mon[29133]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-03-19T03:13:33.054 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:32 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/2131937619' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T03:13:33.054 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:32 smithi059 ceph-mon[29133]: fsmap cephfs:1 {0=cephfs.smithi059.hmylwa=up:active} 3 up:standby 2024-03-19T03:13:33.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:32 smithi176 ceph-mon[36450]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-03-19T03:13:33.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:32 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/2131937619' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T03:13:33.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:32 smithi176 ceph-mon[36450]: fsmap cephfs:1 {0=cephfs.smithi059.hmylwa=up:active} 3 up:standby 2024-03-19T03:13:33.393 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-19T03:13:33.406 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2024-03-19T03:13:33.406 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-19T03:13:34.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:33 smithi059 ceph-mon[29133]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-19T03:13:34.109 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:13:34.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:33 smithi176 ceph-mon[36450]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-19T03:13:34.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:34 smithi059 ceph-mon[29133]: mds.? [v2:172.21.15.176:6826/4058958323,v1:172.21.15.176:6827/4058958323] up:standby 2024-03-19T03:13:34.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:34 smithi059 ceph-mon[29133]: fsmap cephfs:1 {0=cephfs.smithi059.hmylwa=up:active} 3 up:standby 2024-03-19T03:13:35.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:34 smithi176 ceph-mon[36450]: mds.? [v2:172.21.15.176:6826/4058958323,v1:172.21.15.176:6827/4058958323] up:standby 2024-03-19T03:13:35.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:34 smithi176 ceph-mon[36450]: fsmap cephfs:1 {0=cephfs.smithi059.hmylwa=up:active} 3 up:standby 2024-03-19T03:13:36.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:35 smithi059 ceph-mon[29133]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 2 op/s 2024-03-19T03:13:36.128 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:35 smithi176 ceph-mon[36450]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 2 op/s 2024-03-19T03:13:36.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.851+0000 7f5597fcc700 1 -- 172.21.15.59:0/1945346860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55900662e0 msgr2=0x7f55900ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:36.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.851+0000 7f5597fcc700 1 --2- 172.21.15.59:0/1945346860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55900662e0 0x7f55900ff9c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5580004940 tx=0x7f5580009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:36.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.851+0000 7f5597fcc700 1 -- 172.21.15.59:0/1945346860 shutdown_connections 2024-03-19T03:13:36.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.851+0000 7f5597fcc700 1 --2- 172.21.15.59:0/1945346860 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55900fff00 0x7f5590102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:36.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.851+0000 7f5597fcc700 1 --2- 172.21.15.59:0/1945346860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55900662e0 0x7f55900ff9c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:36.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.851+0000 7f5597fcc700 1 -- 172.21.15.59:0/1945346860 >> 172.21.15.59:0/1945346860 conn(0x7f55900f7430 msgr2=0x7f55900f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:36.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.851+0000 7f5597fcc700 1 -- 172.21.15.59:0/1945346860 shutdown_connections 2024-03-19T03:13:36.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.851+0000 7f5597fcc700 1 -- 172.21.15.59:0/1945346860 wait complete. 2024-03-19T03:13:36.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.852+0000 7f5597fcc700 1 Processor -- start 2024-03-19T03:13:36.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.852+0000 7f5597fcc700 1 -- start start 2024-03-19T03:13:36.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.852+0000 7f5597fcc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55900662e0 0x7f55900fd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:36.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.852+0000 7f5597fcc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55900fff00 0x7f55900fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:36.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.852+0000 7f5597fcc700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55900fbe70 con 0x7f55900662e0 2024-03-19T03:13:36.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.852+0000 7f5597fcc700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55900fbfe0 con 0x7f55900fff00 2024-03-19T03:13:36.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.852+0000 7f5595d68700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55900662e0 0x7f55900fd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:36.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.852+0000 7f5595d68700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55900662e0 0x7f55900fd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35348/0 (socket says 172.21.15.59:35348) 2024-03-19T03:13:36.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.852+0000 7f5595d68700 1 -- 172.21.15.59:0/2874764297 learned_addr learned my addr 172.21.15.59:0/2874764297 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:13:36.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.852+0000 7f5595d68700 1 -- 172.21.15.59:0/2874764297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55900fff00 msgr2=0x7f55900fb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:36.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.852+0000 7f5595d68700 1 --2- 172.21.15.59:0/2874764297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55900fff00 0x7f55900fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:36.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.852+0000 7f5595d68700 1 -- 172.21.15.59:0/2874764297 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5580005040 con 0x7f55900662e0 2024-03-19T03:13:36.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.852+0000 7f5595d68700 1 --2- 172.21.15.59:0/2874764297 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55900662e0 0x7f55900fd3f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f558000b3b0 tx=0x7f5580009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:36.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.852+0000 7f55867fc700 1 -- 172.21.15.59:0/2874764297 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f558000e040 con 0x7f55900662e0 2024-03-19T03:13:36.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.853+0000 7f55867fc700 1 -- 172.21.15.59:0/2874764297 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f558000bcc0 con 0x7f55900662e0 2024-03-19T03:13:36.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.853+0000 7f5597fcc700 1 -- 172.21.15.59:0/2874764297 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55900fc260 con 0x7f55900662e0 2024-03-19T03:13:36.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.853+0000 7f55867fc700 1 -- 172.21.15.59:0/2874764297 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f558001c630 con 0x7f55900662e0 2024-03-19T03:13:36.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.853+0000 7f5597fcc700 1 -- 172.21.15.59:0/2874764297 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5590115a60 con 0x7f55900662e0 2024-03-19T03:13:36.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.854+0000 7f5597fcc700 1 -- 172.21.15.59:0/2874764297 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f559005f5f0 con 0x7f55900662e0 2024-03-19T03:13:36.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.854+0000 7f55867fc700 1 -- 172.21.15.59:0/2874764297 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f55800139f0 con 0x7f55900662e0 2024-03-19T03:13:36.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.854+0000 7f55867fc700 1 --2- 172.21.15.59:0/2874764297 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f557c0692a0 0x7f557c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:36.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.854+0000 7f55867fc700 1 -- 172.21.15.59:0/2874764297 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f5580088030 con 0x7f55900662e0 2024-03-19T03:13:36.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.854+0000 7f5595567700 1 --2- 172.21.15.59:0/2874764297 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f557c0692a0 0x7f557c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:36.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.855+0000 7f5595567700 1 --2- 172.21.15.59:0/2874764297 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f557c0692a0 0x7f557c06b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f558c004b70 tx=0x7f558c007de0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:36.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:36.859+0000 7f55867fc700 1 -- 172.21.15.59:0/2874764297 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f55800b9e50 con 0x7f55900662e0 2024-03-19T03:13:37.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:37.106+0000 7f5597fcc700 1 -- 172.21.15.59:0/2874764297 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5590115cb0 con 0x7f55900662e0 2024-03-19T03:13:37.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:37.106+0000 7f55867fc700 1 -- 172.21.15.59:0/2874764297 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1806 (secure 0 0 0) 0x7f55800542c0 con 0x7f55900662e0 2024-03-19T03:13:37.107 INFO:teuthology.orchestra.run.smithi059.stdout:e14 2024-03-19T03:13:37.107 INFO:teuthology.orchestra.run.smithi059.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-19T03:13:37.107 INFO:teuthology.orchestra.run.smithi059.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-19T03:13:37.107 INFO:teuthology.orchestra.run.smithi059.stdout:legacy client fscid: 1 2024-03-19T03:13:37.107 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:13:37.107 INFO:teuthology.orchestra.run.smithi059.stdout:Filesystem 'cephfs' (1) 2024-03-19T03:13:37.107 INFO:teuthology.orchestra.run.smithi059.stdout:fs_name cephfs 2024-03-19T03:13:37.108 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 13 2024-03-19T03:13:37.108 INFO:teuthology.orchestra.run.smithi059.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-19T03:13:37.108 INFO:teuthology.orchestra.run.smithi059.stdout:created 2024-03-19T03:13:17.039133+0000 2024-03-19T03:13:37.108 INFO:teuthology.orchestra.run.smithi059.stdout:modified 2024-03-19T03:13:32.433876+0000 2024-03-19T03:13:37.108 INFO:teuthology.orchestra.run.smithi059.stdout:tableserver 0 2024-03-19T03:13:37.108 INFO:teuthology.orchestra.run.smithi059.stdout:root 0 2024-03-19T03:13:37.108 INFO:teuthology.orchestra.run.smithi059.stdout:session_timeout 60 2024-03-19T03:13:37.108 INFO:teuthology.orchestra.run.smithi059.stdout:session_autoclose 300 2024-03-19T03:13:37.108 INFO:teuthology.orchestra.run.smithi059.stdout:max_file_size 1099511627776 2024-03-19T03:13:37.108 INFO:teuthology.orchestra.run.smithi059.stdout:required_client_features {} 2024-03-19T03:13:37.108 INFO:teuthology.orchestra.run.smithi059.stdout:last_failure 0 2024-03-19T03:13:37.108 INFO:teuthology.orchestra.run.smithi059.stdout:last_failure_osd_epoch 0 2024-03-19T03:13:37.108 INFO:teuthology.orchestra.run.smithi059.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-19T03:13:37.108 INFO:teuthology.orchestra.run.smithi059.stdout:max_mds 1 2024-03-19T03:13:37.108 INFO:teuthology.orchestra.run.smithi059.stdout:in 0 2024-03-19T03:13:37.108 INFO:teuthology.orchestra.run.smithi059.stdout:up {0=14468} 2024-03-19T03:13:37.109 INFO:teuthology.orchestra.run.smithi059.stdout:failed 2024-03-19T03:13:37.109 INFO:teuthology.orchestra.run.smithi059.stdout:damaged 2024-03-19T03:13:37.109 INFO:teuthology.orchestra.run.smithi059.stdout:stopped 2024-03-19T03:13:37.109 INFO:teuthology.orchestra.run.smithi059.stdout:data_pools [3] 2024-03-19T03:13:37.109 INFO:teuthology.orchestra.run.smithi059.stdout:metadata_pool 2 2024-03-19T03:13:37.109 INFO:teuthology.orchestra.run.smithi059.stdout:inline_data enabled 2024-03-19T03:13:37.109 INFO:teuthology.orchestra.run.smithi059.stdout:balancer 2024-03-19T03:13:37.109 INFO:teuthology.orchestra.run.smithi059.stdout:standby_count_wanted 1 2024-03-19T03:13:37.109 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi059.hmylwa{0:14468} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.59:6826/2936511711,v1:172.21.15.59:6827/2936511711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-19T03:13:37.109 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:13:37.109 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:13:37.109 INFO:teuthology.orchestra.run.smithi059.stdout:Standby daemons: 2024-03-19T03:13:37.109 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:13:37.109 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi059.doeuxg{-1:14482} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.59:6828/503726080,v1:172.21.15.59:6829/503726080] compat {c=[1],r=[1],i=[7ff]}] 2024-03-19T03:13:37.109 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi176.bochhq{-1:24279} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.176:6824/193753714,v1:172.21.15.176:6825/193753714] compat {c=[1],r=[1],i=[7ff]}] 2024-03-19T03:13:37.110 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi176.gisnqc{-1:24289} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.176:6826/4058958323,v1:172.21.15.176:6827/4058958323] compat {c=[1],r=[1],i=[7ff]}] 2024-03-19T03:13:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:37.108+0000 7f5597fcc700 1 -- 172.21.15.59:0/2874764297 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f557c0692a0 msgr2=0x7f557c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:37.108+0000 7f5597fcc700 1 --2- 172.21.15.59:0/2874764297 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f557c0692a0 0x7f557c06b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f558c004b70 tx=0x7f558c007de0 comp rx=0 tx=0).stop 2024-03-19T03:13:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:37.108+0000 7f5597fcc700 1 -- 172.21.15.59:0/2874764297 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55900662e0 msgr2=0x7f55900fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:37.108+0000 7f5597fcc700 1 --2- 172.21.15.59:0/2874764297 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55900662e0 0x7f55900fd3f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f558000b3b0 tx=0x7f5580009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:37.108+0000 7f5597fcc700 1 -- 172.21.15.59:0/2874764297 shutdown_connections 2024-03-19T03:13:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:37.108+0000 7f5597fcc700 1 --2- 172.21.15.59:0/2874764297 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f557c0692a0 0x7f557c06b750 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:37.108+0000 7f5597fcc700 1 --2- 172.21.15.59:0/2874764297 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55900662e0 0x7f55900fd3f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:37.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:37.108+0000 7f5597fcc700 1 --2- 172.21.15.59:0/2874764297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55900fff00 0x7f55900fb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:37.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:37.108+0000 7f5597fcc700 1 -- 172.21.15.59:0/2874764297 >> 172.21.15.59:0/2874764297 conn(0x7f55900f7430 msgr2=0x7f55900f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:37.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:37.108+0000 7f5597fcc700 1 -- 172.21.15.59:0/2874764297 shutdown_connections 2024-03-19T03:13:37.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:37.108+0000 7f5597fcc700 1 -- 172.21.15.59:0/2874764297 wait complete. 2024-03-19T03:13:37.112 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 14 2024-03-19T03:13:37.734 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-03-19T03:13:38.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:38 smithi059 ceph-mon[29133]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-19T03:13:38.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:38.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:38.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:13:38.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:13:38.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:38.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:38 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/2874764297' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-19T03:13:38.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:38 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:13:38.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:38 smithi176 ceph-mon[36450]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-19T03:13:38.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:38.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:38.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:13:38.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:13:38.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:38.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:38 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/2874764297' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-19T03:13:38.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:38 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:13:38.575 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:13:40.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:40 smithi059 ceph-mon[29133]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-03-19T03:13:40.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:40 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:40.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:40 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:40.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:40 smithi176 ceph-mon[36450]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-03-19T03:13:40.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:40 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:40.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:40 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:40.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.769+0000 7fe48d61f700 1 -- 172.21.15.59:0/2029976230 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4880fdfe0 msgr2=0x7fe4880fe450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:40.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.769+0000 7fe48d61f700 1 --2- 172.21.15.59:0/2029976230 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4880fdfe0 0x7fe4880fe450 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe470004940 tx=0x7fe470009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:40.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.769+0000 7fe48d61f700 1 -- 172.21.15.59:0/2029976230 shutdown_connections 2024-03-19T03:13:40.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.769+0000 7fe48d61f700 1 --2- 172.21.15.59:0/2029976230 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4880fdfe0 0x7fe4880fe450 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:40.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.769+0000 7fe48d61f700 1 --2- 172.21.15.59:0/2029976230 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4880fce70 0x7fe4880fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:40.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.769+0000 7fe48d61f700 1 -- 172.21.15.59:0/2029976230 >> 172.21.15.59:0/2029976230 conn(0x7fe4880f8550 msgr2=0x7fe4880fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:40.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.769+0000 7fe48d61f700 1 -- 172.21.15.59:0/2029976230 shutdown_connections 2024-03-19T03:13:40.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.769+0000 7fe48d61f700 1 -- 172.21.15.59:0/2029976230 wait complete. 2024-03-19T03:13:40.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.770+0000 7fe48d61f700 1 Processor -- start 2024-03-19T03:13:40.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.770+0000 7fe48d61f700 1 -- start start 2024-03-19T03:13:40.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.770+0000 7fe48d61f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4880fce70 0x7fe48810d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:40.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.770+0000 7fe48d61f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4880fdfe0 0x7fe48810de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:40.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.770+0000 7fe48d61f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48810e500 con 0x7fe4880fdfe0 2024-03-19T03:13:40.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.770+0000 7fe48d61f700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe488112ea0 con 0x7fe4880fce70 2024-03-19T03:13:40.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.770+0000 7fe4867fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4880fdfe0 0x7fe48810de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:40.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.770+0000 7fe4867fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4880fdfe0 0x7fe48810de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35366/0 (socket says 172.21.15.59:35366) 2024-03-19T03:13:40.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.770+0000 7fe4867fc700 1 -- 172.21.15.59:0/108647450 learned_addr learned my addr 172.21.15.59:0/108647450 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:13:40.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.770+0000 7fe486ffd700 1 --2- 172.21.15.59:0/108647450 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4880fce70 0x7fe48810d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:40.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.770+0000 7fe4867fc700 1 -- 172.21.15.59:0/108647450 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4880fce70 msgr2=0x7fe48810d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:40.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.770+0000 7fe4867fc700 1 --2- 172.21.15.59:0/108647450 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4880fce70 0x7fe48810d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:40.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.770+0000 7fe4867fc700 1 -- 172.21.15.59:0/108647450 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe470005040 con 0x7fe4880fdfe0 2024-03-19T03:13:40.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.770+0000 7fe486ffd700 1 --2- 172.21.15.59:0/108647450 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4880fce70 0x7fe48810d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:13:40.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.771+0000 7fe4867fc700 1 --2- 172.21.15.59:0/108647450 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4880fdfe0 0x7fe48810de70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe47000b3b0 tx=0x7fe470009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:40.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.771+0000 7fe47f7fe700 1 -- 172.21.15.59:0/108647450 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe47000e040 con 0x7fe4880fdfe0 2024-03-19T03:13:40.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.771+0000 7fe47f7fe700 1 -- 172.21.15.59:0/108647450 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe47000bcc0 con 0x7fe4880fdfe0 2024-03-19T03:13:40.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.771+0000 7fe48d61f700 1 -- 172.21.15.59:0/108647450 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4881130a0 con 0x7fe4880fdfe0 2024-03-19T03:13:40.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.771+0000 7fe47f7fe700 1 -- 172.21.15.59:0/108647450 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe47001c630 con 0x7fe4880fdfe0 2024-03-19T03:13:40.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.771+0000 7fe48d61f700 1 -- 172.21.15.59:0/108647450 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe488113610 con 0x7fe4880fdfe0 2024-03-19T03:13:40.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.772+0000 7fe48d61f700 1 -- 172.21.15.59:0/108647450 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe48804c710 con 0x7fe4880fdfe0 2024-03-19T03:13:40.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.772+0000 7fe47f7fe700 1 -- 172.21.15.59:0/108647450 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe470013980 con 0x7fe4880fdfe0 2024-03-19T03:13:40.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.773+0000 7fe47f7fe700 1 --2- 172.21.15.59:0/108647450 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe4740692a0 0x7fe47406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:40.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.773+0000 7fe47f7fe700 1 -- 172.21.15.59:0/108647450 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fe470087fc0 con 0x7fe4880fdfe0 2024-03-19T03:13:40.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.773+0000 7fe486ffd700 1 --2- 172.21.15.59:0/108647450 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe4740692a0 0x7fe47406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:40.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.773+0000 7fe486ffd700 1 --2- 172.21.15.59:0/108647450 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe4740692a0 0x7fe47406b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe480001370 tx=0x7fe4800096a0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:40.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:40.777+0000 7fe47f7fe700 1 -- 172.21.15.59:0/108647450 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fe470057d00 con 0x7fe4880fdfe0 2024-03-19T03:13:41.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:41.020+0000 7fe48d61f700 1 -- 172.21.15.59:0/108647450 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe488105950 con 0x7fe4880fdfe0 2024-03-19T03:13:41.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:41.021+0000 7fe47f7fe700 1 -- 172.21.15.59:0/108647450 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4686 (secure 0 0 0) 0x7fe470057890 con 0x7fe4880fdfe0 2024-03-19T03:13:41.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:41.023+0000 7fe48d61f700 1 -- 172.21.15.59:0/108647450 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe4740692a0 msgr2=0x7fe47406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:41.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:41.023+0000 7fe48d61f700 1 --2- 172.21.15.59:0/108647450 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe4740692a0 0x7fe47406b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe480001370 tx=0x7fe4800096a0 comp rx=0 tx=0).stop 2024-03-19T03:13:41.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:41.023+0000 7fe48d61f700 1 -- 172.21.15.59:0/108647450 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4880fdfe0 msgr2=0x7fe48810de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:41.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:41.023+0000 7fe48d61f700 1 --2- 172.21.15.59:0/108647450 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4880fdfe0 0x7fe48810de70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe47000b3b0 tx=0x7fe470009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:41.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:41.023+0000 7fe48d61f700 1 -- 172.21.15.59:0/108647450 shutdown_connections 2024-03-19T03:13:41.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:41.023+0000 7fe48d61f700 1 --2- 172.21.15.59:0/108647450 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe4740692a0 0x7fe47406b750 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:41.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:41.023+0000 7fe48d61f700 1 --2- 172.21.15.59:0/108647450 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4880fce70 0x7fe48810d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:41.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:41.023+0000 7fe48d61f700 1 --2- 172.21.15.59:0/108647450 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4880fdfe0 0x7fe48810de70 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:41.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:41.023+0000 7fe48d61f700 1 -- 172.21.15.59:0/108647450 >> 172.21.15.59:0/108647450 conn(0x7fe4880f8550 msgr2=0x7fe4880f9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:41.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:41.023+0000 7fe48d61f700 1 -- 172.21.15.59:0/108647450 shutdown_connections 2024-03-19T03:13:41.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:41.023+0000 7fe48d61f700 1 -- 172.21.15.59:0/108647450 wait complete. 2024-03-19T03:13:41.026 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 14 2024-03-19T03:13:41.035 INFO:teuthology.orchestra.run.smithi059.stdout:true 2024-03-19T03:13:41.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:41 smithi176 ceph-mon[36450]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-03-19T03:13:41.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:41 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/108647450' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:13:41.693 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:41 smithi059 ceph-mon[29133]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-03-19T03:13:41.693 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:41 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/108647450' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:13:41.718 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-03-19T03:13:42.073 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:13:44.277 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:44 smithi059 ceph-mon[29133]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:44.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:44 smithi176 ceph-mon[36450]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:44.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.954+0000 7f001587f700 1 -- 172.21.15.59:0/891415240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00100fe450 msgr2=0x7f00100fe860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:44.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.954+0000 7f001587f700 1 --2- 172.21.15.59:0/891415240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00100fe450 0x7f00100fe860 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0000004940 tx=0x7f0000009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:44.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.954+0000 7f001587f700 1 -- 172.21.15.59:0/891415240 shutdown_connections 2024-03-19T03:13:44.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.954+0000 7f001587f700 1 --2- 172.21.15.59:0/891415240 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00100feda0 0x7f00100faee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:44.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.954+0000 7f001587f700 1 --2- 172.21.15.59:0/891415240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00100fe450 0x7f00100fe860 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:44.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.954+0000 7f001587f700 1 -- 172.21.15.59:0/891415240 >> 172.21.15.59:0/891415240 conn(0x7f00100f6c10 msgr2=0x7f00100f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:44.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.955+0000 7f001587f700 1 -- 172.21.15.59:0/891415240 shutdown_connections 2024-03-19T03:13:44.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.955+0000 7f001587f700 1 -- 172.21.15.59:0/891415240 wait complete. 2024-03-19T03:13:44.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.955+0000 7f001587f700 1 Processor -- start 2024-03-19T03:13:44.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.955+0000 7f001587f700 1 -- start start 2024-03-19T03:13:44.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.956+0000 7f001587f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00100fe450 0x7f001010d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:44.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.956+0000 7f001587f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00100feda0 0x7f001010de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:44.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.956+0000 7f001587f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001010e490 con 0x7f00100feda0 2024-03-19T03:13:44.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.956+0000 7f001587f700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001010e600 con 0x7f00100fe450 2024-03-19T03:13:44.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.956+0000 7f000e7fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00100feda0 0x7f001010de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:44.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.956+0000 7f000e7fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00100feda0 0x7f001010de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40922/0 (socket says 172.21.15.59:40922) 2024-03-19T03:13:44.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.956+0000 7f000e7fc700 1 -- 172.21.15.59:0/779422545 learned_addr learned my addr 172.21.15.59:0/779422545 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:13:44.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.956+0000 7f000effd700 1 --2- 172.21.15.59:0/779422545 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00100fe450 0x7f001010d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:44.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.956+0000 7f000e7fc700 1 -- 172.21.15.59:0/779422545 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00100fe450 msgr2=0x7f001010d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:44.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.956+0000 7f000e7fc700 1 --2- 172.21.15.59:0/779422545 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00100fe450 0x7f001010d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:44.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.956+0000 7f000e7fc700 1 -- 172.21.15.59:0/779422545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0000005040 con 0x7f00100feda0 2024-03-19T03:13:44.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.956+0000 7f000e7fc700 1 --2- 172.21.15.59:0/779422545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00100feda0 0x7f001010de90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f000400c830 tx=0x7f0004004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:44.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.956+0000 7efff77fe700 1 -- 172.21.15.59:0/779422545 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00040106f0 con 0x7f00100feda0 2024-03-19T03:13:44.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.957+0000 7f001587f700 1 -- 172.21.15.59:0/779422545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0010113060 con 0x7f00100feda0 2024-03-19T03:13:44.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.957+0000 7efff77fe700 1 -- 172.21.15.59:0/779422545 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0004010d20 con 0x7f00100feda0 2024-03-19T03:13:44.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.957+0000 7efff77fe700 1 -- 172.21.15.59:0/779422545 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00040113a0 con 0x7f00100feda0 2024-03-19T03:13:44.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.957+0000 7f001587f700 1 -- 172.21.15.59:0/779422545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0010113620 con 0x7f00100feda0 2024-03-19T03:13:44.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.958+0000 7f001587f700 1 -- 172.21.15.59:0/779422545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f001004c710 con 0x7f00100feda0 2024-03-19T03:13:44.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.958+0000 7efff77fe700 1 -- 172.21.15.59:0/779422545 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0004012400 con 0x7f00100feda0 2024-03-19T03:13:44.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.959+0000 7efff77fe700 1 --2- 172.21.15.59:0/779422545 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7efff80692a0 0x7efff806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:44.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.959+0000 7efff77fe700 1 -- 172.21.15.59:0/779422545 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f0004087150 con 0x7f00100feda0 2024-03-19T03:13:44.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.959+0000 7f000effd700 1 --2- 172.21.15.59:0/779422545 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7efff80692a0 0x7efff806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:44.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.959+0000 7f000effd700 1 --2- 172.21.15.59:0/779422545 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7efff80692a0 0x7efff806b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f000000b970 tx=0x7f0000009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:44.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:44.963+0000 7efff77fe700 1 -- 172.21.15.59:0/779422545 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f0004056f10 con 0x7f00100feda0 2024-03-19T03:13:45.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:45.251+0000 7f001587f700 1 -- 172.21.15.59:0/779422545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f00100fc2b0 con 0x7f00100feda0 2024-03-19T03:13:45.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:45.251+0000 7efff77fe700 1 -- 172.21.15.59:0/779422545 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v14) v1 ==== 78+0+99 (secure 0 0 0) 0x7f0004056aa0 con 0x7f00100feda0 2024-03-19T03:13:45.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:45.253+0000 7f001587f700 1 -- 172.21.15.59:0/779422545 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7efff80692a0 msgr2=0x7efff806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:45.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:45.253+0000 7f001587f700 1 --2- 172.21.15.59:0/779422545 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7efff80692a0 0x7efff806b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f000000b970 tx=0x7f0000009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:45.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:45.253+0000 7f001587f700 1 -- 172.21.15.59:0/779422545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00100feda0 msgr2=0x7f001010de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:45.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:45.253+0000 7f001587f700 1 --2- 172.21.15.59:0/779422545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00100feda0 0x7f001010de90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f000400c830 tx=0x7f0004004470 comp rx=0 tx=0).stop 2024-03-19T03:13:45.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:45.253+0000 7f001587f700 1 -- 172.21.15.59:0/779422545 shutdown_connections 2024-03-19T03:13:45.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:45.253+0000 7f001587f700 1 --2- 172.21.15.59:0/779422545 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7efff80692a0 0x7efff806b750 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:45.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:45.253+0000 7f001587f700 1 --2- 172.21.15.59:0/779422545 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00100fe450 0x7f001010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:45.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:45.253+0000 7f001587f700 1 --2- 172.21.15.59:0/779422545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00100feda0 0x7f001010de90 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:45.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:45.253+0000 7f001587f700 1 -- 172.21.15.59:0/779422545 >> 172.21.15.59:0/779422545 conn(0x7f00100f6c10 msgr2=0x7f00100f82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:45.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:45.253+0000 7f001587f700 1 -- 172.21.15.59:0/779422545 shutdown_connections 2024-03-19T03:13:45.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:45.254+0000 7f001587f700 1 -- 172.21.15.59:0/779422545 wait complete. 2024-03-19T03:13:45.266 INFO:teuthology.orchestra.run.smithi059.stdout:true 2024-03-19T03:13:45.872 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-03-19T03:13:45.886 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:13:46.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.020+0000 7f749006e640 1 -- 172.21.15.59:0/3981118467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74880fc6c0 msgr2=0x7f74880fcad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:46.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.020+0000 7f749006e640 1 --2- 172.21.15.59:0/3981118467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74880fc6c0 0x7f74880fcad0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f74840048d0 tx=0x7f7484009da0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.020+0000 7f749006e640 1 -- 172.21.15.59:0/3981118467 shutdown_connections 2024-03-19T03:13:46.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.020+0000 7f749006e640 1 --2- 172.21.15.59:0/3981118467 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74880fd010 0x7f74880fb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.020+0000 7f749006e640 1 --2- 172.21.15.59:0/3981118467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74880fc6c0 0x7f74880fcad0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.020+0000 7f749006e640 1 -- 172.21.15.59:0/3981118467 >> 172.21.15.59:0/3981118467 conn(0x7f74880f6000 msgr2=0x7f74880f8430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:46.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.020+0000 7f749006e640 1 -- 172.21.15.59:0/3981118467 shutdown_connections 2024-03-19T03:13:46.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.020+0000 7f749006e640 1 -- 172.21.15.59:0/3981118467 wait complete. 2024-03-19T03:13:46.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.021+0000 7f749006e640 1 Processor -- start 2024-03-19T03:13:46.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.021+0000 7f749006e640 1 -- start start 2024-03-19T03:13:46.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.021+0000 7f749006e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74880fc6c0 0x7f748810bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:46.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.021+0000 7f749006e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74880fd010 0x7f748810c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:46.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.021+0000 7f749006e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748810c790 con 0x7f74880fd010 2024-03-19T03:13:46.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.021+0000 7f749006e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7488111190 con 0x7f74880fc6c0 2024-03-19T03:13:46.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.021+0000 7f748d609640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74880fd010 0x7f748810c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:46.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.021+0000 7f748d609640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74880fd010 0x7f748810c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40940/0 (socket says 172.21.15.59:40940) 2024-03-19T03:13:46.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.021+0000 7f748d609640 1 -- 172.21.15.59:0/4148316648 learned_addr learned my addr 172.21.15.59:0/4148316648 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:13:46.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.021+0000 7f748d609640 1 -- 172.21.15.59:0/4148316648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74880fc6c0 msgr2=0x7f748810bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:46.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.021+0000 7f748de0a640 1 --2- 172.21.15.59:0/4148316648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74880fc6c0 0x7f748810bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:46.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.021+0000 7f748d609640 1 --2- 172.21.15.59:0/4148316648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74880fc6c0 0x7f748810bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.021+0000 7f748d609640 1 -- 172.21.15.59:0/4148316648 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7484005040 con 0x7f74880fd010 2024-03-19T03:13:46.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.021+0000 7f748d609640 1 --2- 172.21.15.59:0/4148316648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74880fd010 0x7f748810c140 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f747c00eb70 tx=0x7f747c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:46.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.022+0000 7f747a7fc640 1 -- 172.21.15.59:0/4148316648 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f747c007c30 con 0x7f74880fd010 2024-03-19T03:13:46.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.022+0000 7f747a7fc640 1 -- 172.21.15.59:0/4148316648 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f747c00f040 con 0x7f74880fd010 2024-03-19T03:13:46.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.022+0000 7f749006e640 1 -- 172.21.15.59:0/4148316648 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74881112d0 con 0x7f74880fd010 2024-03-19T03:13:46.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.022+0000 7f747a7fc640 1 -- 172.21.15.59:0/4148316648 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f747c0095b0 con 0x7f74880fd010 2024-03-19T03:13:46.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.022+0000 7f749006e640 1 -- 172.21.15.59:0/4148316648 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74881117c0 con 0x7f74880fd010 2024-03-19T03:13:46.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.023+0000 7f749006e640 1 -- 172.21.15.59:0/4148316648 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74881055c0 con 0x7f74880fd010 2024-03-19T03:13:46.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.023+0000 7f747a7fc640 1 -- 172.21.15.59:0/4148316648 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f747c009710 con 0x7f74880fd010 2024-03-19T03:13:46.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.023+0000 7f747a7fc640 1 --2- 172.21.15.59:0/4148316648 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7474069210 0x7f747406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:46.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.024+0000 7f748de0a640 1 --2- 172.21.15.59:0/4148316648 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7474069210 0x7f747406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:46.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.024+0000 7f747a7fc640 1 -- 172.21.15.59:0/4148316648 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f747c086e30 con 0x7f74880fd010 2024-03-19T03:13:46.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.024+0000 7f748de0a640 1 --2- 172.21.15.59:0/4148316648 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7474069210 0x7f747406b6c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f748402bda0 tx=0x7f7484009da0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:46.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.028+0000 7f747a7fc640 1 -- 172.21.15.59:0/4148316648 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f747c053160 con 0x7f74880fd010 2024-03-19T03:13:46.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.200+0000 7f749006e640 1 -- 172.21.15.59:0/4148316648 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7488103c30 con 0x7f74880fd010 2024-03-19T03:13:46.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.201+0000 7f747a7fc640 1 -- 172.21.15.59:0/4148316648 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+14266 (secure 0 0 0) 0x7f747c056780 con 0x7f74880fd010 2024-03-19T03:13:46.203 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:13:46.203 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":45,"fsid":"8c8ae530-e59d-11ee-95c9-87774f69a715","created":"2024-03-19T03:06:15.064131+0000","modified":"2024-03-19T03:13:19.045790+0000","last_up_change":"2024-03-19T03:11:58.260172+0000","last_in_change":"2024-03-19T03:11:40.023566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:10:45.238939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-19T03:13:15.389389+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-19T03:13:16.280778+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6a12d03-3dd9-40d4-b54e-4b2041c18e48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1770897334},{"type":"v1","addr":"172.21.15.59:6803","nonce":1770897334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1770897334},{"type":"v1","addr":"172.21.15.59:6805","nonce":1770897334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1770897334},{"type":"v1","addr":"172.21.15.59:6809","nonce":1770897334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1770897334},{"type":"v1","addr":"172.21.15.59:6807","nonce":1770897334}]},"public_addr":"172.21.15.59:6803/1770897334","cluster_addr":"172.21.15.59:6805/1770897334","heartbeat_back_addr":"172.21.15.59:6809/1770897334","heartbeat_front_addr":"172.21.15.59:6807/1770897334","state":["exists","up"]},{"osd":1,"uuid":"c1069548-eb23-476b-a661-b71f0a73987e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3595398218},{"type":"v1","addr":"172.21.15.59:6811","nonce":3595398218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3595398218},{"type":"v1","addr":"172.21.15.59:6813","nonce":3595398218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":3595398218},{"type":"v1","addr":"172.21.15.59:6817","nonce":3595398218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3595398218},{"type":"v1","addr":"172.21.15.59:6815","nonce":3595398218}]},"public_addr":"172.21.15.59:6811/3595398218","cluster_addr":"172.21.15.59:6813/3595398218","heartbeat_back_addr":"172.21.15.59:6817/3595398218","heartbeat_front_addr":"172.21.15.59:6815/3595398218","state":["exists","up"]},{"osd":2,"uuid":"4958f1e4-2dc9-458f-82ff-e51d98762cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":639205676},{"type":"v1","addr":"172.21.15.59:6819","nonce":639205676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":639205676},{"type":"v1","addr":"172.21.15.59:6821","nonce":639205676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":639205676},{"type":"v1","addr":"172.21.15.59:6825","nonce":639205676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":639205676},{"type":"v1","addr":"172.21.15.59:6823","nonce":639205676}]},"public_addr":"172.21.15.59:6819/639205676","cluster_addr":"172.21.15.59:6821/639205676","heartbeat_back_addr":"172.21.15.59:6825/639205676","heartbeat_front_addr":"172.21.15.59:6823/639205676","state":["exists","up"]},{"osd":3,"uuid":"c3cf9095-7ce2-4d0f-bc65-2cda09504d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":3291292674},{"type":"v1","addr":"172.21.15.176:6801","nonce":3291292674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":3291292674},{"type":"v1","addr":"172.21.15.176:6803","nonce":3291292674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":3291292674},{"type":"v1","addr":"172.21.15.176:6807","nonce":3291292674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":3291292674},{"type":"v1","addr":"172.21.15.176:6805","nonce":3291292674}]},"public_addr":"172.21.15.176:6801/3291292674","cluster_addr":"172.21.15.176:6803/3291292674","heartbeat_back_addr":"172.21.15.176:6807/3291292674","heartbeat_front_addr":"172.21.15.176:6805/3291292674","state":["exists","up"]},{"osd":4,"uuid":"b102d232-9f65-4d96-ad81-b3d1a0b8faf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":1006246702},{"type":"v1","addr":"172.21.15.176:6809","nonce":1006246702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":1006246702},{"type":"v1","addr":"172.21.15.176:6811","nonce":1006246702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":1006246702},{"type":"v1","addr":"172.21.15.176:6815","nonce":1006246702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":1006246702},{"type":"v1","addr":"172.21.15.176:6813","nonce":1006246702}]},"public_addr":"172.21.15.176:6809/1006246702","cluster_addr":"172.21.15.176:6811/1006246702","heartbeat_back_addr":"172.21.15.176:6815/1006246702","heartbeat_front_addr":"172.21.15.176:6813/1006246702","state":["exists","up"]},{"osd":5,"uuid":"f9c805ed-8aa0-4f39-a78f-c6a4b018eb7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":3279585260},{"type":"v1","addr":"172.21.15.176:6817","nonce":3279585260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":3279585260},{"type":"v1","addr":"172.21.15.176:6819","nonce":3279585260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":3279585260},{"type":"v1","addr":"172.21.15.176:6823","nonce":3279585260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":3279585260},{"type":"v1","addr":"172.21.15.176:6821","nonce":3279585260}]},"public_addr":"172.21.15.176:6817/3279585260","cluster_addr":"172.21.15.176:6819/3279585260","heartbeat_back_addr":"172.21.15.176:6823/3279585260","heartbeat_front_addr":"172.21.15.176:6821/3279585260","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:09:44.576635+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:10:12.913840+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:10:42.022990+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:11:05.039788+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:11:31.198904+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:11:57.135656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:0/2117971082":"2024-03-20T03:08:30.903133+0000","172.21.15.59:0/988882608":"2024-03-20T03:07:20.738216+0000","172.21.15.59:0/2884557489":"2024-03-20T03:08:30.903133+0000","172.21.15.59:6801/1573642037":"2024-03-20T03:07:20.738216+0000","172.21.15.59:0/2956286237":"2024-03-20T03:07:20.738216+0000","172.21.15.59:6801/1857243897":"2024-03-20T03:08:30.903133+0000","172.21.15.59:0/661762888":"2024-03-20T03:06:49.795873+0000","172.21.15.59:0/2659339884":"2024-03-20T03:06:49.795873+0000","172.21.15.59:0/156375080":"2024-03-20T03:06:49.795873+0000","172.21.15.59:6800/3609017287":"2024-03-20T03:06:49.795873+0000","172.21.15.59:0/133172490":"2024-03-20T03:08:30.903133+0000","172.21.15.59:6800/1573642037":"2024-03-20T03:07:20.738216+0000","172.21.15.59:6801/3609017287":"2024-03-20T03:06:49.795873+0000","172.21.15.59:6800/1857243897":"2024-03-20T03:08:30.903133+0000","172.21.15.59:0/2483030805":"2024-03-20T03:07:20.738216+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:13:46.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.203+0000 7f749006e640 1 -- 172.21.15.59:0/4148316648 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7474069210 msgr2=0x7f747406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:46.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.203+0000 7f749006e640 1 --2- 172.21.15.59:0/4148316648 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7474069210 0x7f747406b6c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f748402bda0 tx=0x7f7484009da0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.203+0000 7f749006e640 1 -- 172.21.15.59:0/4148316648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74880fd010 msgr2=0x7f748810c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:46.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.203+0000 7f749006e640 1 --2- 172.21.15.59:0/4148316648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74880fd010 0x7f748810c140 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f747c00eb70 tx=0x7f747c00b040 comp rx=0 tx=0).stop 2024-03-19T03:13:46.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.203+0000 7f749006e640 1 -- 172.21.15.59:0/4148316648 shutdown_connections 2024-03-19T03:13:46.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.203+0000 7f749006e640 1 --2- 172.21.15.59:0/4148316648 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7474069210 0x7f747406b6c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.203+0000 7f749006e640 1 --2- 172.21.15.59:0/4148316648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74880fd010 0x7f748810c140 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.203+0000 7f749006e640 1 --2- 172.21.15.59:0/4148316648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74880fc6c0 0x7f748810bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.203+0000 7f749006e640 1 -- 172.21.15.59:0/4148316648 >> 172.21.15.59:0/4148316648 conn(0x7f74880f6000 msgr2=0x7f74880ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:46.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.203+0000 7f749006e640 1 -- 172.21.15.59:0/4148316648 shutdown_connections 2024-03-19T03:13:46.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.203+0000 7f749006e640 1 -- 172.21.15.59:0/4148316648 wait complete. 2024-03-19T03:13:46.214 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:46 smithi059 ceph-mon[29133]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:46.214 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:46.214 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:46.215 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:13:46.215 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:13:46.215 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:46.215 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:46 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/779422545' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-19T03:13:46.215 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:46.215 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:13:46.216 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T03:10:45.238939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-19T03:13:15.389389+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-19T03:13:16.280778+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T03:13:46.216 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T03:13:46.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.315+0000 7fe1adb5d640 1 -- 172.21.15.59:0/3959222142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80fe140 msgr2=0x7fe1a80fa2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:46.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.315+0000 7fe1adb5d640 1 --2- 172.21.15.59:0/3959222142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80fe140 0x7fe1a80fa2a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe19c0048d0 tx=0x7fe19c009da0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.315+0000 7fe1adb5d640 1 -- 172.21.15.59:0/3959222142 shutdown_connections 2024-03-19T03:13:46.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.315+0000 7fe1adb5d640 1 --2- 172.21.15.59:0/3959222142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80fe140 0x7fe1a80fa2a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.315+0000 7fe1adb5d640 1 --2- 172.21.15.59:0/3959222142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1a80fd7f0 0x7fe1a80fdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.315+0000 7fe1adb5d640 1 -- 172.21.15.59:0/3959222142 >> 172.21.15.59:0/3959222142 conn(0x7fe1a80f5ff0 msgr2=0x7fe1a80f8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:46.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.315+0000 7fe1adb5d640 1 -- 172.21.15.59:0/3959222142 shutdown_connections 2024-03-19T03:13:46.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.315+0000 7fe1adb5d640 1 -- 172.21.15.59:0/3959222142 wait complete. 2024-03-19T03:13:46.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.316+0000 7fe1adb5d640 1 Processor -- start 2024-03-19T03:13:46.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.316+0000 7fe1adb5d640 1 -- start start 2024-03-19T03:13:46.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.316+0000 7fe1adb5d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80fd7f0 0x7fe1a810cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:46.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.316+0000 7fe1adb5d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1a80fe140 0x7fe1a810d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:46.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.316+0000 7fe1adb5d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a810d890 con 0x7fe1a80fd7f0 2024-03-19T03:13:46.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.316+0000 7fe1adb5d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a8112250 con 0x7fe1a80fe140 2024-03-19T03:13:46.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.316+0000 7fe1a77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80fd7f0 0x7fe1a810cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:46.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.316+0000 7fe1a77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80fd7f0 0x7fe1a810cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40964/0 (socket says 172.21.15.59:40964) 2024-03-19T03:13:46.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.316+0000 7fe1a77fe640 1 -- 172.21.15.59:0/1639307866 learned_addr learned my addr 172.21.15.59:0/1639307866 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:13:46.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.316+0000 7fe1a6ffd640 1 --2- 172.21.15.59:0/1639307866 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1a80fe140 0x7fe1a810d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:46.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.316+0000 7fe1a77fe640 1 -- 172.21.15.59:0/1639307866 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1a80fe140 msgr2=0x7fe1a810d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:46.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.316+0000 7fe1a77fe640 1 --2- 172.21.15.59:0/1639307866 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1a80fe140 0x7fe1a810d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.316+0000 7fe1a77fe640 1 -- 172.21.15.59:0/1639307866 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c005040 con 0x7fe1a80fd7f0 2024-03-19T03:13:46.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.317+0000 7fe1a77fe640 1 --2- 172.21.15.59:0/1639307866 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80fd7f0 0x7fe1a810cd00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe198002a50 tx=0x7fe198004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:46.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.317+0000 7fe18ffff640 1 -- 172.21.15.59:0/1639307866 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe198007c30 con 0x7fe1a80fd7f0 2024-03-19T03:13:46.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.317+0000 7fe1adb5d640 1 -- 172.21.15.59:0/1639307866 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a81123f0 con 0x7fe1a80fd7f0 2024-03-19T03:13:46.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.317+0000 7fe18ffff640 1 -- 172.21.15.59:0/1639307866 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe19800f040 con 0x7fe1a80fd7f0 2024-03-19T03:13:46.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.317+0000 7fe18ffff640 1 -- 172.21.15.59:0/1639307866 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe19800db50 con 0x7fe1a80fd7f0 2024-03-19T03:13:46.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.317+0000 7fe1adb5d640 1 -- 172.21.15.59:0/1639307866 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a81128c0 con 0x7fe1a80fd7f0 2024-03-19T03:13:46.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.318+0000 7fe1adb5d640 1 -- 172.21.15.59:0/1639307866 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a804bcc0 con 0x7fe1a80fd7f0 2024-03-19T03:13:46.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.318+0000 7fe18ffff640 1 -- 172.21.15.59:0/1639307866 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe19801c400 con 0x7fe1a80fd7f0 2024-03-19T03:13:46.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.319+0000 7fe18ffff640 1 --2- 172.21.15.59:0/1639307866 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe190069210 0x7fe19006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:46.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.319+0000 7fe18ffff640 1 -- 172.21.15.59:0/1639307866 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fe198086a10 con 0x7fe1a80fd7f0 2024-03-19T03:13:46.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.319+0000 7fe1a6ffd640 1 --2- 172.21.15.59:0/1639307866 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe190069210 0x7fe19006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:46.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.319+0000 7fe1a6ffd640 1 --2- 172.21.15.59:0/1639307866 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe190069210 0x7fe19006b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe19c000c00 tx=0x7fe19c009da0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:46.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.323+0000 7fe18ffff640 1 -- 172.21.15.59:0/1639307866 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fe19804ce50 con 0x7fe1a80fd7f0 2024-03-19T03:13:46.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:46 smithi176 ceph-mon[36450]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:46.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:46 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:46.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:46 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:46.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:46 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:13:46.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:46 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:13:46.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:46 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:46.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:46 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/779422545' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-19T03:13:46.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:46 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:13:46.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:46 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:13:46.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.493+0000 7fe1adb5d640 1 -- 172.21.15.59:0/1639307866 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe1a8112ae0 con 0x7fe1a80fd7f0 2024-03-19T03:13:46.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.494+0000 7fe18ffff640 1 -- 172.21.15.59:0/1639307866 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe1a8112ae0 con 0x7fe1a80fd7f0 2024-03-19T03:13:46.495 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-19T03:13:46.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.495+0000 7fe1adb5d640 1 -- 172.21.15.59:0/1639307866 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe190069210 msgr2=0x7fe19006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:46.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.495+0000 7fe1adb5d640 1 --2- 172.21.15.59:0/1639307866 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe190069210 0x7fe19006b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe19c000c00 tx=0x7fe19c009da0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.495+0000 7fe1adb5d640 1 -- 172.21.15.59:0/1639307866 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80fd7f0 msgr2=0x7fe1a810cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:46.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.495+0000 7fe1adb5d640 1 --2- 172.21.15.59:0/1639307866 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80fd7f0 0x7fe1a810cd00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe198002a50 tx=0x7fe198004600 comp rx=0 tx=0).stop 2024-03-19T03:13:46.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.495+0000 7fe1adb5d640 1 -- 172.21.15.59:0/1639307866 shutdown_connections 2024-03-19T03:13:46.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.495+0000 7fe1adb5d640 1 --2- 172.21.15.59:0/1639307866 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe190069210 0x7fe19006b6c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.495+0000 7fe1adb5d640 1 --2- 172.21.15.59:0/1639307866 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1a80fe140 0x7fe1a810d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.495+0000 7fe1adb5d640 1 --2- 172.21.15.59:0/1639307866 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80fd7f0 0x7fe1a810cd00 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.495+0000 7fe1adb5d640 1 -- 172.21.15.59:0/1639307866 >> 172.21.15.59:0/1639307866 conn(0x7fe1a80f5ff0 msgr2=0x7fe1a80f7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:46.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.496+0000 7fe1adb5d640 1 -- 172.21.15.59:0/1639307866 shutdown_connections 2024-03-19T03:13:46.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.496+0000 7fe1adb5d640 1 -- 172.21.15.59:0/1639307866 wait complete. 2024-03-19T03:13:46.508 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-19T03:13:46.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.606+0000 7ff14f39d640 1 -- 172.21.15.59:0/2267550432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1480fc380 msgr2=0x7ff1480fc7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:46.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.606+0000 7ff14f39d640 1 --2- 172.21.15.59:0/2267550432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1480fc380 0x7ff1480fc7f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff13c0048d0 tx=0x7ff13c009da0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.606+0000 7ff14f39d640 1 -- 172.21.15.59:0/2267550432 shutdown_connections 2024-03-19T03:13:46.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.606+0000 7ff14f39d640 1 --2- 172.21.15.59:0/2267550432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1480fc380 0x7ff1480fc7f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.606+0000 7ff14f39d640 1 --2- 172.21.15.59:0/2267550432 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff1480fb0b0 0x7ff1480fb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.607+0000 7ff14f39d640 1 -- 172.21.15.59:0/2267550432 >> 172.21.15.59:0/2267550432 conn(0x7ff1480f6890 msgr2=0x7ff1480f8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:46.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.607+0000 7ff14f39d640 1 -- 172.21.15.59:0/2267550432 shutdown_connections 2024-03-19T03:13:46.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.607+0000 7ff14f39d640 1 -- 172.21.15.59:0/2267550432 wait complete. 2024-03-19T03:13:46.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.607+0000 7ff14f39d640 1 Processor -- start 2024-03-19T03:13:46.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.607+0000 7ff14f39d640 1 -- start start 2024-03-19T03:13:46.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.607+0000 7ff14f39d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff1480fb0b0 0x7ff14810ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:46.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.607+0000 7ff14f39d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1480fc380 0x7ff14810d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:46.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.607+0000 7ff14f39d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14810d880 con 0x7ff1480fc380 2024-03-19T03:13:46.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.607+0000 7ff14f39d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff148112240 con 0x7ff1480fb0b0 2024-03-19T03:13:46.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.607+0000 7ff14c938640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1480fc380 0x7ff14810d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:46.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.607+0000 7ff14c938640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1480fc380 0x7ff14810d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40996/0 (socket says 172.21.15.59:40996) 2024-03-19T03:13:46.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.607+0000 7ff14c938640 1 -- 172.21.15.59:0/3120697990 learned_addr learned my addr 172.21.15.59:0/3120697990 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:13:46.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.607+0000 7ff14d139640 1 --2- 172.21.15.59:0/3120697990 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff1480fb0b0 0x7ff14810ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:46.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.608+0000 7ff14c938640 1 -- 172.21.15.59:0/3120697990 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff1480fb0b0 msgr2=0x7ff14810ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:46.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.608+0000 7ff14c938640 1 --2- 172.21.15.59:0/3120697990 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff1480fb0b0 0x7ff14810ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.608+0000 7ff14c938640 1 -- 172.21.15.59:0/3120697990 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13c005040 con 0x7ff1480fc380 2024-03-19T03:13:46.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.608+0000 7ff14c938640 1 --2- 172.21.15.59:0/3120697990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1480fc380 0x7ff14810d200 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff13c007d50 tx=0x7ff13c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:46.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.608+0000 7ff145ffb640 1 -- 172.21.15.59:0/3120697990 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff13c009590 con 0x7ff1480fc380 2024-03-19T03:13:46.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.608+0000 7ff145ffb640 1 -- 172.21.15.59:0/3120697990 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff13c0096f0 con 0x7ff1480fc380 2024-03-19T03:13:46.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.608+0000 7ff14f39d640 1 -- 172.21.15.59:0/3120697990 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff148112380 con 0x7ff1480fc380 2024-03-19T03:13:46.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.608+0000 7ff145ffb640 1 -- 172.21.15.59:0/3120697990 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff13c03b6d0 con 0x7ff1480fc380 2024-03-19T03:13:46.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.608+0000 7ff14f39d640 1 -- 172.21.15.59:0/3120697990 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff148112870 con 0x7ff1480fc380 2024-03-19T03:13:46.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.609+0000 7ff14f39d640 1 -- 172.21.15.59:0/3120697990 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14804bcc0 con 0x7ff1480fc380 2024-03-19T03:13:46.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.609+0000 7ff145ffb640 1 -- 172.21.15.59:0/3120697990 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff13c02f070 con 0x7ff1480fc380 2024-03-19T03:13:46.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.610+0000 7ff145ffb640 1 --2- 172.21.15.59:0/3120697990 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7ff130069210 0x7ff13006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:46.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.610+0000 7ff145ffb640 1 -- 172.21.15.59:0/3120697990 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7ff13c0a9700 con 0x7ff1480fc380 2024-03-19T03:13:46.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.610+0000 7ff14d139640 1 --2- 172.21.15.59:0/3120697990 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7ff130069210 0x7ff13006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:46.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.610+0000 7ff14d139640 1 --2- 172.21.15.59:0/3120697990 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7ff130069210 0x7ff13006b6c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff1400041d0 tx=0x7ff140004330 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:46.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.614+0000 7ff145ffb640 1 -- 172.21.15.59:0/3120697990 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7ff13c075a30 con 0x7ff1480fc380 2024-03-19T03:13:46.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.787+0000 7ff14f39d640 1 -- 172.21.15.59:0/3120697990 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7ff148103c00 con 0x7ff1480fc380 2024-03-19T03:13:46.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.788+0000 7ff145ffb640 1 -- 172.21.15.59:0/3120697990 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v45) v1 ==== 107+0+11 (secure 0 0 0) 0x7ff13c079050 con 0x7ff1480fc380 2024-03-19T03:13:46.790 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-19T03:13:46.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.790+0000 7ff14f39d640 1 -- 172.21.15.59:0/3120697990 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7ff130069210 msgr2=0x7ff13006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:46.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.790+0000 7ff14f39d640 1 --2- 172.21.15.59:0/3120697990 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7ff130069210 0x7ff13006b6c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff1400041d0 tx=0x7ff140004330 comp rx=0 tx=0).stop 2024-03-19T03:13:46.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.790+0000 7ff14f39d640 1 -- 172.21.15.59:0/3120697990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1480fc380 msgr2=0x7ff14810d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:46.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.790+0000 7ff14f39d640 1 --2- 172.21.15.59:0/3120697990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1480fc380 0x7ff14810d200 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff13c007d50 tx=0x7ff13c009da0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.790+0000 7ff14f39d640 1 -- 172.21.15.59:0/3120697990 shutdown_connections 2024-03-19T03:13:46.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.790+0000 7ff14f39d640 1 --2- 172.21.15.59:0/3120697990 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7ff130069210 0x7ff13006b6c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.790+0000 7ff14f39d640 1 --2- 172.21.15.59:0/3120697990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1480fc380 0x7ff14810d200 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.790+0000 7ff14f39d640 1 --2- 172.21.15.59:0/3120697990 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff1480fb0b0 0x7ff14810ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.790+0000 7ff14f39d640 1 -- 172.21.15.59:0/3120697990 >> 172.21.15.59:0/3120697990 conn(0x7ff1480f6890 msgr2=0x7ff1480f8320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:46.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.790+0000 7ff14f39d640 1 -- 172.21.15.59:0/3120697990 shutdown_connections 2024-03-19T03:13:46.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.790+0000 7ff14f39d640 1 -- 172.21.15.59:0/3120697990 wait complete. 2024-03-19T03:13:46.802 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-19T03:13:46.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.901+0000 7fb17c55d640 1 -- 172.21.15.59:0/2495690228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1740ff340 msgr2=0x7fb174101720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:46.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.901+0000 7fb17c55d640 1 --2- 172.21.15.59:0/2495690228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1740ff340 0x7fb174101720 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb1640048d0 tx=0x7fb164009da0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.901+0000 7fb17c55d640 1 -- 172.21.15.59:0/2495690228 shutdown_connections 2024-03-19T03:13:46.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.901+0000 7fb17c55d640 1 --2- 172.21.15.59:0/2495690228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1740ff340 0x7fb174101720 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.901+0000 7fb17c55d640 1 --2- 172.21.15.59:0/2495690228 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1740661f0 0x7fb1740fee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.901+0000 7fb17c55d640 1 -- 172.21.15.59:0/2495690228 >> 172.21.15.59:0/2495690228 conn(0x7fb1740f6850 msgr2=0x7fb1740f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:46.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.901+0000 7fb17c55d640 1 -- 172.21.15.59:0/2495690228 shutdown_connections 2024-03-19T03:13:46.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.901+0000 7fb17c55d640 1 -- 172.21.15.59:0/2495690228 wait complete. 2024-03-19T03:13:46.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.902+0000 7fb17c55d640 1 Processor -- start 2024-03-19T03:13:46.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.902+0000 7fb17c55d640 1 -- start start 2024-03-19T03:13:46.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.902+0000 7fb17c55d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1740661f0 0x7fb17410cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:46.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.902+0000 7fb17c55d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1740ff340 0x7fb17410d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:46.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.902+0000 7fb17c55d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17410d920 con 0x7fb1740661f0 2024-03-19T03:13:46.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.902+0000 7fb17c55d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb174112290 con 0x7fb1740ff340 2024-03-19T03:13:46.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.902+0000 7fb17a2f9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1740661f0 0x7fb17410cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:46.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.902+0000 7fb17a2f9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1740661f0 0x7fb17410cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41012/0 (socket says 172.21.15.59:41012) 2024-03-19T03:13:46.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.902+0000 7fb17a2f9640 1 -- 172.21.15.59:0/2681050504 learned_addr learned my addr 172.21.15.59:0/2681050504 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:13:46.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.902+0000 7fb179af8640 1 --2- 172.21.15.59:0/2681050504 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1740ff340 0x7fb17410d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:46.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.902+0000 7fb17a2f9640 1 -- 172.21.15.59:0/2681050504 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1740ff340 msgr2=0x7fb17410d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:46.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.902+0000 7fb17a2f9640 1 --2- 172.21.15.59:0/2681050504 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1740ff340 0x7fb17410d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:46.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.902+0000 7fb17a2f9640 1 -- 172.21.15.59:0/2681050504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb164005040 con 0x7fb1740661f0 2024-03-19T03:13:46.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.902+0000 7fb17a2f9640 1 --2- 172.21.15.59:0/2681050504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1740661f0 0x7fb17410cd90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb170002840 tx=0x7fb170004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:46.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.902+0000 7fb16affd640 1 -- 172.21.15.59:0/2681050504 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb170007c30 con 0x7fb1740661f0 2024-03-19T03:13:46.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.903+0000 7fb16affd640 1 -- 172.21.15.59:0/2681050504 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb17000f040 con 0x7fb1740661f0 2024-03-19T03:13:46.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.903+0000 7fb17c55d640 1 -- 172.21.15.59:0/2681050504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb174112430 con 0x7fb1740661f0 2024-03-19T03:13:46.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.903+0000 7fb16affd640 1 -- 172.21.15.59:0/2681050504 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb170009570 con 0x7fb1740661f0 2024-03-19T03:13:46.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.903+0000 7fb17c55d640 1 -- 172.21.15.59:0/2681050504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb174112900 con 0x7fb1740661f0 2024-03-19T03:13:46.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.903+0000 7fb17c55d640 1 -- 172.21.15.59:0/2681050504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb17404bcc0 con 0x7fb1740661f0 2024-03-19T03:13:46.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.904+0000 7fb16affd640 1 -- 172.21.15.59:0/2681050504 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb17000dc80 con 0x7fb1740661f0 2024-03-19T03:13:46.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.904+0000 7fb16affd640 1 --2- 172.21.15.59:0/2681050504 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb160069210 0x7fb16006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:46.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.904+0000 7fb16affd640 1 -- 172.21.15.59:0/2681050504 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fb170087150 con 0x7fb1740661f0 2024-03-19T03:13:46.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.904+0000 7fb179af8640 1 --2- 172.21.15.59:0/2681050504 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb160069210 0x7fb16006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:46.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.905+0000 7fb179af8640 1 --2- 172.21.15.59:0/2681050504 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb160069210 0x7fb16006b6c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb164009990 tx=0x7fb164009da0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:46.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:46.909+0000 7fb16affd640 1 -- 172.21.15.59:0/2681050504 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fb1700533e0 con 0x7fb1740661f0 2024-03-19T03:13:47.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.081+0000 7fb17c55d640 1 -- 172.21.15.59:0/2681050504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fb174104d10 con 0x7fb1740661f0 2024-03-19T03:13:47.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.082+0000 7fb16affd640 1 -- 172.21.15.59:0/2681050504 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v45) v1 ==== 107+0+11 (secure 0 0 0) 0x7fb170056a00 con 0x7fb1740661f0 2024-03-19T03:13:47.083 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-19T03:13:47.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.084+0000 7fb17c55d640 1 -- 172.21.15.59:0/2681050504 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb160069210 msgr2=0x7fb16006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:47.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.084+0000 7fb17c55d640 1 --2- 172.21.15.59:0/2681050504 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb160069210 0x7fb16006b6c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb164009990 tx=0x7fb164009da0 comp rx=0 tx=0).stop 2024-03-19T03:13:47.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.084+0000 7fb17c55d640 1 -- 172.21.15.59:0/2681050504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1740661f0 msgr2=0x7fb17410cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:47.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.084+0000 7fb17c55d640 1 --2- 172.21.15.59:0/2681050504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1740661f0 0x7fb17410cd90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb170002840 tx=0x7fb170004570 comp rx=0 tx=0).stop 2024-03-19T03:13:47.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.084+0000 7fb17c55d640 1 -- 172.21.15.59:0/2681050504 shutdown_connections 2024-03-19T03:13:47.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.084+0000 7fb17c55d640 1 --2- 172.21.15.59:0/2681050504 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb160069210 0x7fb16006b6c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:47.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.084+0000 7fb17c55d640 1 --2- 172.21.15.59:0/2681050504 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1740ff340 0x7fb17410d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:47.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.084+0000 7fb17c55d640 1 --2- 172.21.15.59:0/2681050504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1740661f0 0x7fb17410cd90 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:47.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.084+0000 7fb17c55d640 1 -- 172.21.15.59:0/2681050504 >> 172.21.15.59:0/2681050504 conn(0x7fb1740f6850 msgr2=0x7fb1740f6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:47.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.084+0000 7fb17c55d640 1 -- 172.21.15.59:0/2681050504 shutdown_connections 2024-03-19T03:13:47.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.084+0000 7fb17c55d640 1 -- 172.21.15.59:0/2681050504 wait complete. 2024-03-19T03:13:47.096 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:13:47.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.193+0000 7f6638cae640 1 -- 172.21.15.59:0/1426250489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66340fb0b0 msgr2=0x7f66340fb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:47.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.193+0000 7f6638cae640 1 --2- 172.21.15.59:0/1426250489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66340fb0b0 0x7f66340fb4c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f66240048d0 tx=0x7f6624009da0 comp rx=0 tx=0).stop 2024-03-19T03:13:47.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.193+0000 7f6638cae640 1 -- 172.21.15.59:0/1426250489 shutdown_connections 2024-03-19T03:13:47.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.193+0000 7f6638cae640 1 --2- 172.21.15.59:0/1426250489 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66340fc380 0x7f66340fc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:47.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.193+0000 7f6638cae640 1 --2- 172.21.15.59:0/1426250489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66340fb0b0 0x7f66340fb4c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:47.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.193+0000 7f6638cae640 1 -- 172.21.15.59:0/1426250489 >> 172.21.15.59:0/1426250489 conn(0x7f66340f6850 msgr2=0x7f66340f8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:47.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.193+0000 7f6638cae640 1 -- 172.21.15.59:0/1426250489 shutdown_connections 2024-03-19T03:13:47.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.193+0000 7f6638cae640 1 -- 172.21.15.59:0/1426250489 wait complete. 2024-03-19T03:13:47.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.194+0000 7f6638cae640 1 Processor -- start 2024-03-19T03:13:47.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.194+0000 7f6638cae640 1 -- start start 2024-03-19T03:13:47.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.194+0000 7f6638cae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66340fb0b0 0x7f663410cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:47.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.194+0000 7f6638cae640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66340fc380 0x7f663410d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:47.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.194+0000 7f6638cae640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663410d8c0 con 0x7f66340fb0b0 2024-03-19T03:13:47.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.194+0000 7f6638cae640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6634112230 con 0x7f66340fc380 2024-03-19T03:13:47.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.194+0000 7f663259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66340fb0b0 0x7f663410cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:47.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.194+0000 7f663259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66340fb0b0 0x7f663410cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41036/0 (socket says 172.21.15.59:41036) 2024-03-19T03:13:47.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.194+0000 7f663259c640 1 -- 172.21.15.59:0/2898068287 learned_addr learned my addr 172.21.15.59:0/2898068287 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:13:47.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.194+0000 7f662bfff640 1 --2- 172.21.15.59:0/2898068287 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66340fc380 0x7f663410d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:47.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.194+0000 7f663259c640 1 -- 172.21.15.59:0/2898068287 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66340fc380 msgr2=0x7f663410d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:47.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.194+0000 7f663259c640 1 --2- 172.21.15.59:0/2898068287 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66340fc380 0x7f663410d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:47.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.194+0000 7f663259c640 1 -- 172.21.15.59:0/2898068287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6624005040 con 0x7f66340fb0b0 2024-03-19T03:13:47.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.194+0000 7f662bfff640 1 --2- 172.21.15.59:0/2898068287 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66340fc380 0x7f663410d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T03:13:47.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.194+0000 7f663259c640 1 --2- 172.21.15.59:0/2898068287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66340fb0b0 0x7f663410cd00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6624007dd0 tx=0x7f6624009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:47.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.194+0000 7f662affd640 1 -- 172.21.15.59:0/2898068287 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66240094c0 con 0x7f66340fb0b0 2024-03-19T03:13:47.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.194+0000 7f662affd640 1 -- 172.21.15.59:0/2898068287 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6624009620 con 0x7f66340fb0b0 2024-03-19T03:13:47.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.194+0000 7f6638cae640 1 -- 172.21.15.59:0/2898068287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6634112370 con 0x7f66340fb0b0 2024-03-19T03:13:47.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.194+0000 7f662affd640 1 -- 172.21.15.59:0/2898068287 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f662403b640 con 0x7f66340fb0b0 2024-03-19T03:13:47.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.194+0000 7f6638cae640 1 -- 172.21.15.59:0/2898068287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6634112860 con 0x7f66340fb0b0 2024-03-19T03:13:47.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.195+0000 7f6638cae640 1 -- 172.21.15.59:0/2898068287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f663404bcc0 con 0x7f66340fb0b0 2024-03-19T03:13:47.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.196+0000 7f662affd640 1 -- 172.21.15.59:0/2898068287 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f662402f070 con 0x7f66340fb0b0 2024-03-19T03:13:47.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.196+0000 7f662affd640 1 --2- 172.21.15.59:0/2898068287 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f6620069210 0x7f662006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:47.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.196+0000 7f662affd640 1 -- 172.21.15.59:0/2898068287 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f66240a9690 con 0x7f66340fb0b0 2024-03-19T03:13:47.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.196+0000 7f662bfff640 1 --2- 172.21.15.59:0/2898068287 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f6620069210 0x7f662006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:47.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.197+0000 7f662bfff640 1 --2- 172.21.15.59:0/2898068287 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f6620069210 0x7f662006b6c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f663410e1a0 tx=0x7f661c00b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:47.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.201+0000 7f662affd640 1 -- 172.21.15.59:0/2898068287 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f66240759c0 con 0x7f66340fb0b0 2024-03-19T03:13:47.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.411+0000 7f6638cae640 1 -- 172.21.15.59:0/2898068287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6634103c00 con 0x7f66340fb0b0 2024-03-19T03:13:47.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.411+0000 7f662affd640 1 -- 172.21.15.59:0/2898068287 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4686 (secure 0 0 0) 0x7f6624078fe0 con 0x7f66340fb0b0 2024-03-19T03:13:47.413 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:13:47.413 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14482,"name":"cephfs.smithi059.doeuxg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.59:6829/503726080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":503726080},{"type":"v1","addr":"172.21.15.59:6829","nonce":503726080}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24279,"name":"cephfs.smithi176.bochhq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.176:6825/193753714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":193753714},{"type":"v1","addr":"172.21.15.176:6825","nonce":193753714}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24289,"name":"cephfs.smithi176.gisnqc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.176:6827/4058958323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":4058958323},{"type":"v1","addr":"172.21.15.176:6827","nonce":4058958323}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T03:13:17.039133+0000","modified":"2024-03-19T03:13:32.433876+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14468},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14468":{"gid":14468,"name":"cephfs.smithi059.hmylwa","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6827/2936511711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2936511711},{"type":"v1","addr":"172.21.15.59:6827","nonce":2936511711}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-03-19T03:13:47.413 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 14 2024-03-19T03:13:47.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.413+0000 7f6638cae640 1 -- 172.21.15.59:0/2898068287 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f6620069210 msgr2=0x7f662006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:47.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.413+0000 7f6638cae640 1 --2- 172.21.15.59:0/2898068287 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f6620069210 0x7f662006b6c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f663410e1a0 tx=0x7f661c00b040 comp rx=0 tx=0).stop 2024-03-19T03:13:47.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.413+0000 7f6638cae640 1 -- 172.21.15.59:0/2898068287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66340fb0b0 msgr2=0x7f663410cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:47.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.413+0000 7f6638cae640 1 --2- 172.21.15.59:0/2898068287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66340fb0b0 0x7f663410cd00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6624007dd0 tx=0x7f6624009da0 comp rx=0 tx=0).stop 2024-03-19T03:13:47.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.413+0000 7f6638cae640 1 -- 172.21.15.59:0/2898068287 shutdown_connections 2024-03-19T03:13:47.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.413+0000 7f6638cae640 1 --2- 172.21.15.59:0/2898068287 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f6620069210 0x7f662006b6c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:47.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.413+0000 7f6638cae640 1 --2- 172.21.15.59:0/2898068287 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66340fc380 0x7f663410d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:47.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.413+0000 7f6638cae640 1 --2- 172.21.15.59:0/2898068287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66340fb0b0 0x7f663410cd00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:47.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.413+0000 7f6638cae640 1 -- 172.21.15.59:0/2898068287 >> 172.21.15.59:0/2898068287 conn(0x7f66340f6850 msgr2=0x7f6634065590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:47.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.413+0000 7f6638cae640 1 -- 172.21.15.59:0/2898068287 shutdown_connections 2024-03-19T03:13:47.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:47.413+0000 7f6638cae640 1 -- 172.21.15.59:0/2898068287 wait complete. 2024-03-19T03:13:47.425 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:47 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/4148316648' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:13:47.425 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:47 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/1639307866' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:13:47.425 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:47 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/3120697990' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-19T03:13:47.425 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:47 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/2681050504' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-19T03:13:47.426 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 13, 'max_mds': 1, 'flags': 18} 2024-03-19T03:13:47.426 INFO:teuthology.run_tasks:Running task kclient... 2024-03-19T03:13:47.471 INFO:tasks.kclient:Mounting kernel clients... 2024-03-19T03:13:47.472 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-03-19T03:13:47.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:13:47.472 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-03-19T03:13:47.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:13:47.489 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link delete ceph-brx 2024-03-19T03:13:47.557 INFO:teuthology.orchestra.run.smithi059.stderr:Cannot find device "ceph-brx" 2024-03-19T03:13:47.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:13:47.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:13:47.559 DEBUG:teuthology.orchestra.run.smithi176:> ip netns list 2024-03-19T03:13:47.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:13:47.582 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link delete ceph-brx 2024-03-19T03:13:47.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:47 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/4148316648' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:13:47.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:47 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/1639307866' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:13:47.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:47 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/3120697990' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-19T03:13:47.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:47 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/2681050504' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-19T03:13:47.663 INFO:teuthology.orchestra.run.smithi176.stderr:Cannot find device "ceph-brx" 2024-03-19T03:13:47.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:13:47.664 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-03-19T03:13:47.664 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-19T03:13:47.665 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-19T03:13:47.665 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-19T03:13:47.926 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:13:48.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:48 smithi059 ceph-mon[29133]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:48.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:48 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/2898068287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:13:48.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:48 smithi176 ceph-mon[36450]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:48.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:48 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/2898068287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:13:49.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.521+0000 7f83951a2700 1 -- 172.21.15.59:0/3905804425 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8390101d70 msgr2=0x7f8390102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:49.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.521+0000 7f83951a2700 1 --2- 172.21.15.59:0/3905804425 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8390101d70 0x7f8390102140 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8380009ce0 tx=0x7f8380004d10 comp rx=0 tx=0).stop 2024-03-19T03:13:49.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.521+0000 7f83951a2700 1 -- 172.21.15.59:0/3905804425 shutdown_connections 2024-03-19T03:13:49.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.521+0000 7f83951a2700 1 --2- 172.21.15.59:0/3905804425 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8390102710 0x7f83900fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:49.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.521+0000 7f83951a2700 1 --2- 172.21.15.59:0/3905804425 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8390101d70 0x7f8390102140 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:49.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.521+0000 7f83951a2700 1 -- 172.21.15.59:0/3905804425 >> 172.21.15.59:0/3905804425 conn(0x7f8390072bc0 msgr2=0x7f8390072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:49.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.521+0000 7f83951a2700 1 -- 172.21.15.59:0/3905804425 shutdown_connections 2024-03-19T03:13:49.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.521+0000 7f83951a2700 1 -- 172.21.15.59:0/3905804425 wait complete. 2024-03-19T03:13:49.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.521+0000 7f83951a2700 1 Processor -- start 2024-03-19T03:13:49.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.521+0000 7f83951a2700 1 -- start start 2024-03-19T03:13:49.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.522+0000 7f83951a2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8390101d70 0x7f839010ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:49.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.522+0000 7f83951a2700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8390102710 0x7f839010f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:49.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.522+0000 7f83951a2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839010f850 con 0x7f8390101d70 2024-03-19T03:13:49.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.522+0000 7f83951a2700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839010f990 con 0x7f8390102710 2024-03-19T03:13:49.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.522+0000 7f838ed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8390101d70 0x7f839010ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:49.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.522+0000 7f838ed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8390101d70 0x7f839010ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41042/0 (socket says 172.21.15.59:41042) 2024-03-19T03:13:49.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.522+0000 7f838ed9d700 1 -- 172.21.15.59:0/3685287088 learned_addr learned my addr 172.21.15.59:0/3685287088 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:13:49.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.522+0000 7f838e59c700 1 --2- 172.21.15.59:0/3685287088 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8390102710 0x7f839010f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:49.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.522+0000 7f838ed9d700 1 -- 172.21.15.59:0/3685287088 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8390102710 msgr2=0x7f839010f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:49.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.522+0000 7f838ed9d700 1 --2- 172.21.15.59:0/3685287088 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8390102710 0x7f839010f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:49.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.522+0000 7f838ed9d700 1 -- 172.21.15.59:0/3685287088 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8380009940 con 0x7f8390101d70 2024-03-19T03:13:49.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.522+0000 7f838ed9d700 1 --2- 172.21.15.59:0/3685287088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8390101d70 0x7f839010ec30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f838000be90 tx=0x7f8380004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:49.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.522+0000 7f83877fe700 1 -- 172.21.15.59:0/3685287088 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f838000e040 con 0x7f8390101d70 2024-03-19T03:13:49.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.522+0000 7f83877fe700 1 -- 172.21.15.59:0/3685287088 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8380004610 con 0x7f8390101d70 2024-03-19T03:13:49.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.522+0000 7f83951a2700 1 -- 172.21.15.59:0/3685287088 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8390117ca0 con 0x7f8390101d70 2024-03-19T03:13:49.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.523+0000 7f83877fe700 1 -- 172.21.15.59:0/3685287088 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f838001c6d0 con 0x7f8390101d70 2024-03-19T03:13:49.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.523+0000 7f83951a2700 1 -- 172.21.15.59:0/3685287088 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8390118230 con 0x7f8390101d70 2024-03-19T03:13:49.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.524+0000 7f83951a2700 1 -- 172.21.15.59:0/3685287088 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83900ff080 con 0x7f8390101d70 2024-03-19T03:13:49.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.524+0000 7f83877fe700 1 -- 172.21.15.59:0/3685287088 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f8380012b00 con 0x7f8390101d70 2024-03-19T03:13:49.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.525+0000 7f83877fe700 1 --2- 172.21.15.59:0/3685287088 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f837c0692a0 0x7f837c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:49.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.525+0000 7f838e59c700 1 --2- 172.21.15.59:0/3685287088 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f837c0692a0 0x7f837c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:49.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.525+0000 7f83877fe700 1 -- 172.21.15.59:0/3685287088 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f8380088c70 con 0x7f8390101d70 2024-03-19T03:13:49.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.525+0000 7f838e59c700 1 --2- 172.21.15.59:0/3685287088 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f837c0692a0 0x7f837c06b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8390066000 tx=0x7f83780091a0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:49.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.529+0000 7f83877fe700 1 -- 172.21.15.59:0/3685287088 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f838008f080 con 0x7f8390101d70 2024-03-19T03:13:49.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.763+0000 7f83951a2700 1 -- 172.21.15.59:0/3685287088 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f839004bed0 con 0x7f8390101d70 2024-03-19T03:13:49.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.764+0000 7f83877fe700 1 -- 172.21.15.59:0/3685287088 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+83 (secure 0 0 0) 0x7f8380054f00 con 0x7f8390101d70 2024-03-19T03:13:49.765 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-19T03:13:49.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.766+0000 7f83951a2700 1 -- 172.21.15.59:0/3685287088 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f837c0692a0 msgr2=0x7f837c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:49.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.766+0000 7f83951a2700 1 --2- 172.21.15.59:0/3685287088 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f837c0692a0 0x7f837c06b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8390066000 tx=0x7f83780091a0 comp rx=0 tx=0).stop 2024-03-19T03:13:49.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.766+0000 7f83951a2700 1 -- 172.21.15.59:0/3685287088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8390101d70 msgr2=0x7f839010ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:49.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.766+0000 7f83951a2700 1 --2- 172.21.15.59:0/3685287088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8390101d70 0x7f839010ec30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f838000be90 tx=0x7f8380004d10 comp rx=0 tx=0).stop 2024-03-19T03:13:49.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.766+0000 7f83951a2700 1 -- 172.21.15.59:0/3685287088 shutdown_connections 2024-03-19T03:13:49.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.766+0000 7f83951a2700 1 --2- 172.21.15.59:0/3685287088 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f837c0692a0 0x7f837c06b750 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:49.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.766+0000 7f83951a2700 1 --2- 172.21.15.59:0/3685287088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8390101d70 0x7f839010ec30 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:49.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.766+0000 7f83951a2700 1 --2- 172.21.15.59:0/3685287088 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8390102710 0x7f839010f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:49.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.766+0000 7f83951a2700 1 -- 172.21.15.59:0/3685287088 >> 172.21.15.59:0/3685287088 conn(0x7f8390072bc0 msgr2=0x7f83900fae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:49.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.766+0000 7f83951a2700 1 -- 172.21.15.59:0/3685287088 shutdown_connections 2024-03-19T03:13:49.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:49.766+0000 7f83951a2700 1 -- 172.21.15.59:0/3685287088 wait complete. 2024-03-19T03:13:50.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:50 smithi059 ceph-mon[29133]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:50.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:50 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/3685287088' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T03:13:50.372 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-19T03:13:50.372 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-19T03:13:50.372 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-03-19T03:13:50.372 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-19T03:13:50.372 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-19T03:13:50.372 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-19T03:13:50.372 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-19T03:13:50.372 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-19T03:13:50.372 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-19T03:13:50.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:13:50.373 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-03-19T03:13:50.389 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-19T03:13:50.389 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-19T03:13:50.389 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-03-19T03:13:50.389 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-19T03:13:50.389 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-03-19T03:13:50.389 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-19T03:13:50.389 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T03:13:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-03-19T03:13:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-03-19T03:13:50.390 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T03:13:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-03-19T03:13:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-03-19T03:13:50.390 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-19T03:13:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-03-19T03:13:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-03-19T03:13:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-19T03:13:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 40819sec preferred_lft 40819sec 2024-03-19T03:13:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-03-19T03:13:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-19T03:13:50.390 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T03:13:50.391 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-03-19T03:13:50.391 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-03-19T03:13:50.391 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-19T03:13:50.391 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T03:13:50.391 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-19T03:13:50.391 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link add name ceph-brx type bridge 2024-03-19T03:13:50.391 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip addr flush dev ceph-brx 2024-03-19T03:13:50.391 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set ceph-brx up 2024-03-19T03:13:50.391 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-19T03:13:50.392 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-19T03:13:50.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:13:50.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:13:50.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:13:50.600 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-19T03:13:50.626 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-03-19T03:13:50.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:13:50.628 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-03-19T03:13:50.629 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:50 smithi176 ceph-mon[36450]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:50.629 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:50 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/3685287088' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T03:13:50.713 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-03-19T03:13:50.713 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-19T03:13:50.713 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-19T03:13:50.713 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-19T03:13:50.713 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-19T03:13:50.713 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T03:13:50.714 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-19T03:13:50.714 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-19T03:13:50.714 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-19T03:13:50.714 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-19T03:13:50.714 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-19T03:13:50.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:13:50.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:13:50.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:13:50.967 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-03-19T03:13:50.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:13:50.981 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list-id 2024-03-19T03:13:51.036 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T03:13:51.036 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-19T03:13:51.036 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T03:13:51.036 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-19T03:13:51.036 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-19T03:13:51.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:13:51.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:13:51.158 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-19T03:13:51.159 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T03:13:51.159 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-19T03:13:51.159 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-19T03:13:51.159 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-19T03:13:51.159 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-19T03:13:51.159 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-19T03:13:51.159 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-19T03:13:51.159 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-19T03:13:51.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:13:51.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:13:51.292 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T03:13:51.292 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-19T03:13:51.292 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-03-19T03:13:51.292 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set dev brx.0 master ceph-brx 2024-03-19T03:13:51.293 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-19T03:13:51.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:13:51.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:13:51.374 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T03:13:51.375 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-19T03:13:51.389 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-19T03:13:51.389 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T03:13:51.389 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-19T03:13:51.444 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-19T03:13:51.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:13:51.444 DEBUG:teuthology.orchestra.run.smithi059:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-19T03:13:51.548 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-19T03:13:51.549 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-19T03:13:51.549 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-03-19T03:13:51.646 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-19T03:13:51.646 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-19T03:13:51.646 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-03-19T03:13:51.646 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.176:3300" 2024-03-19T03:13:51.646 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-19T03:13:51.651 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-19T03:13:51.651 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-19T03:13:51.651 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-03-19T03:13:51.651 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.176:3300" 2024-03-19T03:13:51.651 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-19T03:13:51.651 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.176:6789" 2024-03-19T03:13:51.651 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.176:6789:/ 2024-03-19T03:13:51.652 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=8c8ae530-e59d-11ee-95c9-87774f69a715" will pass to kernel 2024-03-19T03:13:51.652 INFO:teuthology.orchestra.run.smithi059.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-19T03:13:51.652 INFO:teuthology.orchestra.run.smithi059.stdout: You just mounted a file system that supports labels which does not 2024-03-19T03:13:51.652 INFO:teuthology.orchestra.run.smithi059.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-19T03:13:51.652 INFO:teuthology.orchestra.run.smithi059.stdout: applications will generate AVC messages and not be allowed access to 2024-03-19T03:13:51.652 INFO:teuthology.orchestra.run.smithi059.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-19T03:13:51.654 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-19T03:13:51.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:13:51.654 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-19T03:13:51.677 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:51 smithi059 ceph-mon[29133]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:51.681 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-19T03:13:51.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:51 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-19T03:13:51.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:13:51.785 INFO:teuthology.orchestra.run.smithi059.stdout:client14530 2024-03-19T03:13:51.785 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-19T03:13:51.785 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T03:13:51.802 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-19T03:13:51.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:51 smithi176 ceph-mon[36450]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:13:51.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:51 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-19T03:13:51.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:13:51.905 INFO:teuthology.orchestra.run.smithi059.stdout:client14530 2024-03-19T03:13:51.905 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-19T03:13:51.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:51 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-19T03:13:51.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:13:51.968 INFO:teuthology.orchestra.run.smithi059.stdout:client14530 2024-03-19T03:13:51.968 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715.client14530/status') 2024-03-19T03:13:52.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:13:52.034 INFO:teuthology.orchestra.run.smithi059.stdout:instance: client.14530 (3)192.168.0.1:0/2977583688 2024-03-19T03:13:52.034 INFO:teuthology.orchestra.run.smithi059.stdout:blocklisted: false 2024-03-19T03:13:52.035 INFO:teuthology.orchestra.run.smithi059.stderr:0+1 records in 2024-03-19T03:13:52.035 INFO:teuthology.orchestra.run.smithi059.stderr:0+1 records out 2024-03-19T03:13:52.035 INFO:teuthology.orchestra.run.smithi059.stderr:70 bytes copied, 2.0874e-05 s, 3.4 MB/s 2024-03-19T03:13:52.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:13:52.042 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-03-19T03:13:52.042 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-19T03:13:52.042 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-19T03:13:52.042 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-19T03:13:52.304 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:13:54.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.077+0000 7f79810e0700 1 -- 172.21.15.59:0/3257546555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f797c102e00 msgr2=0x7f797c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:54.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.077+0000 7f79810e0700 1 --2- 172.21.15.59:0/3257546555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f797c102e00 0x7f797c1031d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f796c009ce0 tx=0x7f796c004d10 comp rx=0 tx=0).stop 2024-03-19T03:13:54.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.077+0000 7f79810e0700 1 -- 172.21.15.59:0/3257546555 shutdown_connections 2024-03-19T03:13:54.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.077+0000 7f79810e0700 1 --2- 172.21.15.59:0/3257546555 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f797c1037a0 0x7f797c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:54.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.077+0000 7f79810e0700 1 --2- 172.21.15.59:0/3257546555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f797c102e00 0x7f797c1031d0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:54.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.077+0000 7f79810e0700 1 -- 172.21.15.59:0/3257546555 >> 172.21.15.59:0/3257546555 conn(0x7f797c0f89b0 msgr2=0x7f797c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:54.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.077+0000 7f79810e0700 1 -- 172.21.15.59:0/3257546555 shutdown_connections 2024-03-19T03:13:54.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.077+0000 7f79810e0700 1 -- 172.21.15.59:0/3257546555 wait complete. 2024-03-19T03:13:54.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.078+0000 7f79810e0700 1 Processor -- start 2024-03-19T03:13:54.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.078+0000 7f79810e0700 1 -- start start 2024-03-19T03:13:54.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.078+0000 7f79810e0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f797c102e00 0x7f797c113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:54.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.078+0000 7f79810e0700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f797c1037a0 0x7f797c114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:54.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.078+0000 7f79810e0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797c114680 con 0x7f797c102e00 2024-03-19T03:13:54.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.078+0000 7f79810e0700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797c10dcb0 con 0x7f797c1037a0 2024-03-19T03:13:54.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.078+0000 7f797ad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f797c102e00 0x7f797c113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:54.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.078+0000 7f797ad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f797c102e00 0x7f797c113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50304/0 (socket says 172.21.15.59:50304) 2024-03-19T03:13:54.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.078+0000 7f797ad9d700 1 -- 172.21.15.59:0/4127497280 learned_addr learned my addr 172.21.15.59:0/4127497280 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:13:54.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.078+0000 7f797a59c700 1 --2- 172.21.15.59:0/4127497280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f797c1037a0 0x7f797c114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:54.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.079+0000 7f797ad9d700 1 -- 172.21.15.59:0/4127497280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f797c1037a0 msgr2=0x7f797c114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:54.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.079+0000 7f797ad9d700 1 --2- 172.21.15.59:0/4127497280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f797c1037a0 0x7f797c114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:54.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.079+0000 7f797ad9d700 1 -- 172.21.15.59:0/4127497280 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796c009940 con 0x7f797c102e00 2024-03-19T03:13:54.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.079+0000 7f797ad9d700 1 --2- 172.21.15.59:0/4127497280 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f797c102e00 0x7f797c113af0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f796c003400 tx=0x7f796c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:54.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.079+0000 7f79737fe700 1 -- 172.21.15.59:0/4127497280 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f796c00e040 con 0x7f797c102e00 2024-03-19T03:13:54.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.079+0000 7f79810e0700 1 -- 172.21.15.59:0/4127497280 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f797c10df30 con 0x7f797c102e00 2024-03-19T03:13:54.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.079+0000 7f79737fe700 1 -- 172.21.15.59:0/4127497280 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f796c004610 con 0x7f797c102e00 2024-03-19T03:13:54.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.079+0000 7f79737fe700 1 -- 172.21.15.59:0/4127497280 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f796c01c640 con 0x7f797c102e00 2024-03-19T03:13:54.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.079+0000 7f79810e0700 1 -- 172.21.15.59:0/4127497280 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f797c10e4f0 con 0x7f797c102e00 2024-03-19T03:13:54.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.080+0000 7f79810e0700 1 -- 172.21.15.59:0/4127497280 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f797c100110 con 0x7f797c102e00 2024-03-19T03:13:54.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.080+0000 7f79737fe700 1 -- 172.21.15.59:0/4127497280 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f796c012a70 con 0x7f797c102e00 2024-03-19T03:13:54.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.081+0000 7f79737fe700 1 --2- 172.21.15.59:0/4127497280 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f79680692a0 0x7f796806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:54.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.081+0000 7f79737fe700 1 -- 172.21.15.59:0/4127497280 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f796c088c50 con 0x7f797c102e00 2024-03-19T03:13:54.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.081+0000 7f797a59c700 1 --2- 172.21.15.59:0/4127497280 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f79680692a0 0x7f796806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:54.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.082+0000 7f797a59c700 1 --2- 172.21.15.59:0/4127497280 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f79680692a0 0x7f796806b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f797c10f390 tx=0x7f796400b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:54.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.086+0000 7f79737fe700 1 -- 172.21.15.59:0/4127497280 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f796c058990 con 0x7f797c102e00 2024-03-19T03:13:54.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.319+0000 7f79810e0700 1 -- 172.21.15.59:0/4127497280 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f797c109f00 con 0x7f797c102e00 2024-03-19T03:13:54.321 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:54 smithi059 ceph-mon[29133]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-19T03:13:54.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.320+0000 7f79737fe700 1 -- 172.21.15.59:0/4127497280 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+83 (secure 0 0 0) 0x7f796c004e10 con 0x7f797c102e00 2024-03-19T03:13:54.322 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-19T03:13:54.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.323+0000 7f79810e0700 1 -- 172.21.15.59:0/4127497280 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f79680692a0 msgr2=0x7f796806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:54.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.323+0000 7f79810e0700 1 --2- 172.21.15.59:0/4127497280 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f79680692a0 0x7f796806b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f797c10f390 tx=0x7f796400b040 comp rx=0 tx=0).stop 2024-03-19T03:13:54.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.323+0000 7f79810e0700 1 -- 172.21.15.59:0/4127497280 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f797c102e00 msgr2=0x7f797c113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:54.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.323+0000 7f79810e0700 1 --2- 172.21.15.59:0/4127497280 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f797c102e00 0x7f797c113af0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f796c003400 tx=0x7f796c004d10 comp rx=0 tx=0).stop 2024-03-19T03:13:54.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.323+0000 7f79810e0700 1 -- 172.21.15.59:0/4127497280 shutdown_connections 2024-03-19T03:13:54.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.323+0000 7f79810e0700 1 --2- 172.21.15.59:0/4127497280 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f79680692a0 0x7f796806b750 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:54.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.323+0000 7f79810e0700 1 --2- 172.21.15.59:0/4127497280 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f797c102e00 0x7f797c113af0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:54.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.323+0000 7f79810e0700 1 --2- 172.21.15.59:0/4127497280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f797c1037a0 0x7f797c114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:54.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.323+0000 7f79810e0700 1 -- 172.21.15.59:0/4127497280 >> 172.21.15.59:0/4127497280 conn(0x7f797c0f89b0 msgr2=0x7f797c1073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:54.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.323+0000 7f79810e0700 1 -- 172.21.15.59:0/4127497280 shutdown_connections 2024-03-19T03:13:54.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:54.323+0000 7f79810e0700 1 -- 172.21.15.59:0/4127497280 wait complete. 2024-03-19T03:13:54.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:54 smithi176 ceph-mon[36450]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-19T03:13:54.968 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-19T03:13:54.968 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-19T03:13:54.968 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi176.front.sepia.ceph.com 2024-03-19T03:13:54.968 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-19T03:13:54.969 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-19T03:13:54.969 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-19T03:13:54.969 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-19T03:13:54.969 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-19T03:13:54.969 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-19T03:13:54.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:13:54.969 DEBUG:teuthology.orchestra.run.smithi176:> ip addr 2024-03-19T03:13:54.988 INFO:teuthology.orchestra.run.smithi176.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-19T03:13:54.988 INFO:teuthology.orchestra.run.smithi176.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-19T03:13:54.988 INFO:teuthology.orchestra.run.smithi176.stdout: inet 127.0.0.1/8 scope host lo 2024-03-19T03:13:54.988 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-03-19T03:13:54.988 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 ::1/128 scope host 2024-03-19T03:13:54.988 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-03-19T03:13:54.988 INFO:teuthology.orchestra.run.smithi176.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T03:13:54.988 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:80 brd ff:ff:ff:ff:ff:ff 2024-03-19T03:13:54.988 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f0 2024-03-19T03:13:54.988 INFO:teuthology.orchestra.run.smithi176.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T03:13:54.988 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:81 brd ff:ff:ff:ff:ff:ff 2024-03-19T03:13:54.989 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f1 2024-03-19T03:13:54.989 INFO:teuthology.orchestra.run.smithi176.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T03:13:54.989 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:60 brd ff:ff:ff:ff:ff:ff 2024-03-19T03:13:54.989 INFO:teuthology.orchestra.run.smithi176.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-19T03:13:54.989 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:61 brd ff:ff:ff:ff:ff:ff 2024-03-19T03:13:54.989 INFO:teuthology.orchestra.run.smithi176.stdout: inet 172.21.15.176/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-19T03:13:54.989 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft 40493sec preferred_lft 40493sec 2024-03-19T03:13:54.989 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::ec4:7aff:fe88:6f61/64 scope link 2024-03-19T03:13:54.989 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-03-19T03:13:54.989 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-19T03:13:54.990 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T03:13:54.990 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-03-19T03:13:54.990 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link add name ceph-brx type bridge 2024-03-19T03:13:54.990 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip addr flush dev ceph-brx 2024-03-19T03:13:54.990 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link set ceph-brx up 2024-03-19T03:13:54.990 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-19T03:13:54.990 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-03-19T03:13:55.084 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:13:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:13:55.234 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:13:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:13:55.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:13:55.241 DEBUG:teuthology.orchestra.run.smithi176:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-19T03:13:55.268 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-03-19T03:13:55.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:13:55.269 DEBUG:teuthology.orchestra.run.smithi176:> route 2024-03-19T03:13:55.279 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:55 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/4127497280' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T03:13:55.335 INFO:teuthology.orchestra.run.smithi176.stdout:Kernel IP routing table 2024-03-19T03:13:55.335 INFO:teuthology.orchestra.run.smithi176.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-19T03:13:55.335 INFO:teuthology.orchestra.run.smithi176.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-19T03:13:55.335 INFO:teuthology.orchestra.run.smithi176.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-19T03:13:55.335 INFO:teuthology.orchestra.run.smithi176.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-19T03:13:55.335 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T03:13:55.336 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-03-19T03:13:55.336 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-19T03:13:55.336 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-19T03:13:55.336 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-19T03:13:55.336 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-03-19T03:13:55.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:55 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/4127497280' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T03:13:55.384 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:13:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:13:55.582 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:13:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:13:55.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:13:55.588 DEBUG:teuthology.orchestra.run.smithi176:> ip netns list 2024-03-19T03:13:55.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:13:55.604 DEBUG:teuthology.orchestra.run.smithi176:> ip netns list-id 2024-03-19T03:13:55.659 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T03:13:55.659 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-03-19T03:13:55.659 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-19T03:13:55.659 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-19T03:13:55.659 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-03-19T03:13:55.747 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:13:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:13:55.777 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:13:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:13:55.783 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-19T03:13:55.784 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T03:13:55.784 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-03-19T03:13:55.784 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-19T03:13:55.784 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-19T03:13:55.784 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-19T03:13:55.784 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-19T03:13:55.784 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-19T03:13:55.784 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-03-19T03:13:55.831 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:13:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:13:55.924 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:13:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:13:55.931 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T03:13:55.931 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-03-19T03:13:55.931 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link set brx.0 up 2024-03-19T03:13:55.931 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link set dev brx.0 master ceph-brx 2024-03-19T03:13:55.931 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-03-19T03:13:55.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:13:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:13:56.007 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:13:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:13:56.013 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T03:13:56.013 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-19T03:13:56.027 INFO:teuthology.orchestra.run.smithi176.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-19T03:13:56.027 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T03:13:56.027 DEBUG:teuthology.orchestra.run.smithi176:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-19T03:13:56.082 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-19T03:13:56.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:13:56.082 DEBUG:teuthology.orchestra.run.smithi176:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-19T03:13:56.127 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:56 smithi176 ceph-mon[36450]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-19T03:13:56.178 INFO:teuthology.orchestra.run.smithi176.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-19T03:13:56.178 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-19T03:13:56.178 INFO:teuthology.orchestra.run.smithi176.stdout:invalid new device string format 2024-03-19T03:13:56.278 INFO:teuthology.orchestra.run.smithi176.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-19T03:13:56.278 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-19T03:13:56.278 INFO:teuthology.orchestra.run.smithi176.stdout:invalid new device string format 2024-03-19T03:13:56.279 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.176:3300" 2024-03-19T03:13:56.279 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-19T03:13:56.289 INFO:teuthology.orchestra.run.smithi176.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-19T03:13:56.289 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-19T03:13:56.289 INFO:teuthology.orchestra.run.smithi176.stdout:invalid new device string format 2024-03-19T03:13:56.289 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.176:3300" 2024-03-19T03:13:56.290 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-19T03:13:56.290 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.176:6789" 2024-03-19T03:13:56.290 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.176:6789:/ 2024-03-19T03:13:56.290 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=8c8ae530-e59d-11ee-95c9-87774f69a715" will pass to kernel 2024-03-19T03:13:56.290 INFO:teuthology.orchestra.run.smithi176.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-19T03:13:56.290 INFO:teuthology.orchestra.run.smithi176.stdout: You just mounted a file system that supports labels which does not 2024-03-19T03:13:56.290 INFO:teuthology.orchestra.run.smithi176.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-19T03:13:56.290 INFO:teuthology.orchestra.run.smithi176.stdout: applications will generate AVC messages and not be allowed access to 2024-03-19T03:13:56.291 INFO:teuthology.orchestra.run.smithi176.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-19T03:13:56.293 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-19T03:13:56.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:13:56.293 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-19T03:13:56.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:56 smithi059 ceph-mon[29133]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-19T03:13:56.360 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-19T03:13:56.448 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:13:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-19T03:13:56.449 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:13:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:13:56.456 INFO:teuthology.orchestra.run.smithi176.stdout:client14536 2024-03-19T03:13:56.456 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-19T03:13:56.456 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T03:13:56.472 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-19T03:13:56.559 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:13:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-19T03:13:56.561 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:13:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:13:56.566 INFO:teuthology.orchestra.run.smithi176.stdout:client14536 2024-03-19T03:13:56.566 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-19T03:13:56.614 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:13:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-19T03:13:56.616 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:13:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:13:56.621 INFO:teuthology.orchestra.run.smithi176.stdout:client14536 2024-03-19T03:13:56.622 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715.client14536/status') 2024-03-19T03:13:56.669 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:13:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:13:56.682 INFO:teuthology.orchestra.run.smithi176.stdout:instance: client.14536 (3)172.21.15.176:0/3200515323 2024-03-19T03:13:56.682 INFO:teuthology.orchestra.run.smithi176.stdout:blocklisted: false 2024-03-19T03:13:56.682 INFO:teuthology.orchestra.run.smithi176.stderr:0+1 records in 2024-03-19T03:13:56.682 INFO:teuthology.orchestra.run.smithi176.stderr:0+1 records out 2024-03-19T03:13:56.682 INFO:teuthology.orchestra.run.smithi176.stderr:72 bytes copied, 2.1264e-05 s, 3.4 MB/s 2024-03-19T03:13:56.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-19T03:13:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:13:56.689 INFO:teuthology.run_tasks:Running task print... 2024-03-19T03:13:56.700 INFO:teuthology.task.print:**** done client 2024-03-19T03:13:56.700 INFO:teuthology.run_tasks:Running task parallel... 2024-03-19T03:13:56.753 INFO:teuthology.task.parallel:starting parallel... 2024-03-19T03:13:56.753 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-19T03:13:56.755 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-19T03:13:56.755 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2024-03-19T03:13:56.756 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -e sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs false || true' 2024-03-19T03:13:56.756 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-19T03:13:56.756 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-03-19T03:13:56.760 INFO:tasks.workunit:Pulling workunits from ref e48b1f6c0608b2615e1890cc46286c3877d6e910 2024-03-19T03:13:56.760 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-19T03:13:56.760 INFO:tasks.workunit:timeout=3h 2024-03-19T03:13:56.760 INFO:tasks.workunit:cleanup=True 2024-03-19T03:13:56.761 DEBUG:teuthology.orchestra.run.smithi059:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-19T03:13:56.780 INFO:teuthology.orchestra.run.smithi059.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-19T03:13:56.781 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-19T03:13:56.781 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 94h/148d Inode: 1 Links: 2 2024-03-19T03:13:56.781 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:13:56.781 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-19T03:13:56.781 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-19T03:13:56.781 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-03-19 03:13:21.406003117 +0000 2024-03-19T03:13:56.781 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-03-19 03:13:51.678859022 +0000 2024-03-19T03:13:56.781 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-03-19 03:13:21.406003117 +0000 2024-03-19T03:13:56.781 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-19T03:13:56.781 DEBUG:teuthology.orchestra.run.smithi059:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-19T03:13:56.854 DEBUG:teuthology.orchestra.run.smithi176:> stat -- /home/ubuntu/cephtest/mnt.1 2024-03-19T03:13:56.870 INFO:teuthology.orchestra.run.smithi176.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-03-19T03:13:56.870 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-03-19T03:13:56.870 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 73h/115d Inode: 1 Links: 3 2024-03-19T03:13:56.870 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:13:56.870 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-19T03:13:56.871 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-19T03:13:56.871 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-19 03:13:56.847791065 +0000 2024-03-19T03:13:56.871 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-19 03:13:56.847791065 +0000 2024-03-19T03:13:56.871 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-19 03:13:21.406003117 +0000 2024-03-19T03:13:56.871 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-03-19T03:13:56.871 DEBUG:teuthology.orchestra.run.smithi176:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-03-19T03:13:56.943 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e48b1f6c0608b2615e1890cc46286c3877d6e910 2024-03-19T03:13:56.943 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout e48b1f6c0608b2615e1890cc46286c3877d6e910 2024-03-19T03:13:56.962 INFO:tasks.workunit.client.0.smithi059.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-19T03:13:56.999 INFO:tasks.workunit.client.1.smithi176.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-03-19T03:13:57.043 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:13:58.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:13:58 smithi059 ceph-mon[29133]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-19T03:13:58.379 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:13:58 smithi176 ceph-mon[36450]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-19T03:13:58.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.775+0000 7fbf61415700 1 -- 172.21.15.59:0/2642600222 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf5c066b70 msgr2=0x7fbf5c0ff9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:58.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.775+0000 7fbf61415700 1 --2- 172.21.15.59:0/2642600222 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf5c066b70 0x7fbf5c0ff9d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c004940 tx=0x7fbf4c009e30 comp rx=0 tx=0).stop 2024-03-19T03:13:58.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.775+0000 7fbf61415700 1 -- 172.21.15.59:0/2642600222 shutdown_connections 2024-03-19T03:13:58.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.775+0000 7fbf61415700 1 --2- 172.21.15.59:0/2642600222 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf5c0fff10 0x7fbf5c102330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:58.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.775+0000 7fbf61415700 1 --2- 172.21.15.59:0/2642600222 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf5c066b70 0x7fbf5c0ff9d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:58.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.775+0000 7fbf61415700 1 -- 172.21.15.59:0/2642600222 >> 172.21.15.59:0/2642600222 conn(0x7fbf5c0f7440 msgr2=0x7fbf5c0f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:13:58.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.775+0000 7fbf61415700 1 -- 172.21.15.59:0/2642600222 shutdown_connections 2024-03-19T03:13:58.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.775+0000 7fbf61415700 1 -- 172.21.15.59:0/2642600222 wait complete. 2024-03-19T03:13:58.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.775+0000 7fbf61415700 1 Processor -- start 2024-03-19T03:13:58.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.775+0000 7fbf61415700 1 -- start start 2024-03-19T03:13:58.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.776+0000 7fbf61415700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf5c066b70 0x7fbf5c10da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:58.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.776+0000 7fbf61415700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf5c0fff10 0x7fbf5c10dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:58.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.776+0000 7fbf61415700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c10e5d0 con 0x7fbf5c0fff10 2024-03-19T03:13:58.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.776+0000 7fbf61415700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c10ed80 con 0x7fbf5c066b70 2024-03-19T03:13:58.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.776+0000 7fbf5a7fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf5c0fff10 0x7fbf5c10dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:58.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.776+0000 7fbf5a7fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf5c0fff10 0x7fbf5c10dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50330/0 (socket says 172.21.15.59:50330) 2024-03-19T03:13:58.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.776+0000 7fbf5a7fc700 1 -- 172.21.15.59:0/1333422229 learned_addr learned my addr 172.21.15.59:0/1333422229 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:13:58.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.776+0000 7fbf5affd700 1 --2- 172.21.15.59:0/1333422229 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf5c066b70 0x7fbf5c10da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:58.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.776+0000 7fbf5a7fc700 1 -- 172.21.15.59:0/1333422229 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf5c066b70 msgr2=0x7fbf5c10da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:13:58.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.776+0000 7fbf5a7fc700 1 --2- 172.21.15.59:0/1333422229 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf5c066b70 0x7fbf5c10da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:13:58.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.776+0000 7fbf5a7fc700 1 -- 172.21.15.59:0/1333422229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4c005040 con 0x7fbf5c0fff10 2024-03-19T03:13:58.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.776+0000 7fbf5a7fc700 1 --2- 172.21.15.59:0/1333422229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf5c0fff10 0x7fbf5c10dfd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbf5000eb70 tx=0x7fbf5000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:58.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.776+0000 7fbf437fe700 1 -- 172.21.15.59:0/1333422229 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf50007cc0 con 0x7fbf5c0fff10 2024-03-19T03:13:58.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.777+0000 7fbf61415700 1 -- 172.21.15.59:0/1333422229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5c10f000 con 0x7fbf5c0fff10 2024-03-19T03:13:58.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.777+0000 7fbf437fe700 1 -- 172.21.15.59:0/1333422229 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf5000f040 con 0x7fbf5c0fff10 2024-03-19T03:13:58.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.777+0000 7fbf437fe700 1 -- 172.21.15.59:0/1333422229 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf5000ca90 con 0x7fbf5c0fff10 2024-03-19T03:13:58.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.777+0000 7fbf61415700 1 -- 172.21.15.59:0/1333422229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5c10f540 con 0x7fbf5c0fff10 2024-03-19T03:13:58.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.778+0000 7fbf61415700 1 -- 172.21.15.59:0/1333422229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf5c05f5f0 con 0x7fbf5c0fff10 2024-03-19T03:13:58.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.778+0000 7fbf437fe700 1 -- 172.21.15.59:0/1333422229 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fbf5000cbf0 con 0x7fbf5c0fff10 2024-03-19T03:13:58.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.779+0000 7fbf437fe700 1 --2- 172.21.15.59:0/1333422229 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fbf440692a0 0x7fbf4406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:13:58.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.779+0000 7fbf437fe700 1 -- 172.21.15.59:0/1333422229 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fbf50086c80 con 0x7fbf5c0fff10 2024-03-19T03:13:58.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.779+0000 7fbf5affd700 1 --2- 172.21.15.59:0/1333422229 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fbf440692a0 0x7fbf4406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:13:58.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.779+0000 7fbf5affd700 1 --2- 172.21.15.59:0/1333422229 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fbf440692a0 0x7fbf4406b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c004890 tx=0x7fbf4c009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:13:58.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.783+0000 7fbf437fe700 1 -- 172.21.15.59:0/1333422229 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fbf50056a40 con 0x7fbf5c0fff10 2024-03-19T03:13:58.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.976+0000 7fbf61415700 1 -- 172.21.15.59:0/1333422229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7fbf5c1058f0 con 0x7fbf5c0fff10 2024-03-19T03:13:58.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:13:58.976+0000 7fbf437fe700 1 -- 172.21.15.59:0/1333422229 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=-22 unrecognized config option 'mgr/orchestrator/fail_fs' v21)=-22 unrecognized config option 'mgr/orchestrator/fail_fs' v21) v1 ==== 179+0+0 (secure 0 0 0) 0x7fbf500565d0 con 0x7fbf5c0fff10 2024-03-19T03:13:58.979 INFO:teuthology.orchestra.run.smithi059.stderr:Error EINVAL: unrecognized config option 'mgr/orchestrator/fail_fs' 2024-03-19T03:13:59.608 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-19T03:13:59.608 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2024-03-19T03:13:59.609 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -e sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-19T03:13:59.914 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:14:00.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:00 smithi059 ceph-mon[29133]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-03-19T03:14:00.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:00 smithi176 ceph-mon[36450]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-03-19T03:14:01.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:14:01.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:01 smithi176 ceph-mon[36450]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-03-19T03:14:01.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.648+0000 7fb3396fe700 1 -- 172.21.15.59:0/3845946749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3340fff10 msgr2=0x7fb334102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:01.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.648+0000 7fb3396fe700 1 --2- 172.21.15.59:0/3845946749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3340fff10 0x7fb334102330 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb31c004940 tx=0x7fb31c009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:01.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.648+0000 7fb3396fe700 1 -- 172.21.15.59:0/3845946749 shutdown_connections 2024-03-19T03:14:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.648+0000 7fb3396fe700 1 --2- 172.21.15.59:0/3845946749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3340fff10 0x7fb334102330 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.648+0000 7fb3396fe700 1 --2- 172.21.15.59:0/3845946749 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb334066fd0 0x7fb3340ff9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.648+0000 7fb3396fe700 1 -- 172.21.15.59:0/3845946749 >> 172.21.15.59:0/3845946749 conn(0x7fb3340f7440 msgr2=0x7fb3340f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.648+0000 7fb3396fe700 1 -- 172.21.15.59:0/3845946749 shutdown_connections 2024-03-19T03:14:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.648+0000 7fb3396fe700 1 -- 172.21.15.59:0/3845946749 wait complete. 2024-03-19T03:14:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.649+0000 7fb3396fe700 1 Processor -- start 2024-03-19T03:14:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.649+0000 7fb3396fe700 1 -- start start 2024-03-19T03:14:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.649+0000 7fb3396fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb334066fd0 0x7fb3340fd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.649+0000 7fb3396fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3340fff10 0x7fb3340fb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.649+0000 7fb3396fe700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3340fd7d0 con 0x7fb334066fd0 2024-03-19T03:14:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.649+0000 7fb3396fe700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3340fbe10 con 0x7fb3340fff10 2024-03-19T03:14:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.650+0000 7fb332ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb334066fd0 0x7fb3340fd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.650+0000 7fb332ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb334066fd0 0x7fb3340fd1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50346/0 (socket says 172.21.15.59:50346) 2024-03-19T03:14:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.650+0000 7fb332ffd700 1 -- 172.21.15.59:0/3103666062 learned_addr learned my addr 172.21.15.59:0/3103666062 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:14:01.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.650+0000 7fb3327fc700 1 --2- 172.21.15.59:0/3103666062 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3340fff10 0x7fb3340fb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:01.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.650+0000 7fb332ffd700 1 -- 172.21.15.59:0/3103666062 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3340fff10 msgr2=0x7fb3340fb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:01.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.650+0000 7fb332ffd700 1 --2- 172.21.15.59:0/3103666062 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3340fff10 0x7fb3340fb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:01.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.650+0000 7fb332ffd700 1 -- 172.21.15.59:0/3103666062 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31c005040 con 0x7fb334066fd0 2024-03-19T03:14:01.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.650+0000 7fb332ffd700 1 --2- 172.21.15.59:0/3103666062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb334066fd0 0x7fb3340fd1d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb324002a50 tx=0x7fb3240046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:01.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.650+0000 7fb32b7fe700 1 -- 172.21.15.59:0/3103666062 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb324007cc0 con 0x7fb334066fd0 2024-03-19T03:14:01.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.650+0000 7fb32b7fe700 1 -- 172.21.15.59:0/3103666062 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb32400f040 con 0x7fb334066fd0 2024-03-19T03:14:01.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.650+0000 7fb32b7fe700 1 -- 172.21.15.59:0/3103666062 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb32400db30 con 0x7fb334066fd0 2024-03-19T03:14:01.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.650+0000 7fb3396fe700 1 -- 172.21.15.59:0/3103666062 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3340fc0f0 con 0x7fb334066fd0 2024-03-19T03:14:01.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.650+0000 7fb3396fe700 1 -- 172.21.15.59:0/3103666062 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3340fc5e0 con 0x7fb334066fd0 2024-03-19T03:14:01.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.651+0000 7fb3396fe700 1 -- 172.21.15.59:0/3103666062 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb33404c710 con 0x7fb334066fd0 2024-03-19T03:14:01.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.652+0000 7fb32b7fe700 1 -- 172.21.15.59:0/3103666062 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb32400dc90 con 0x7fb334066fd0 2024-03-19T03:14:01.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.652+0000 7fb32b7fe700 1 --2- 172.21.15.59:0/3103666062 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb3200692a0 0x7fb32006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:01.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.652+0000 7fb32b7fe700 1 -- 172.21.15.59:0/3103666062 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fb324086dd0 con 0x7fb334066fd0 2024-03-19T03:14:01.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.652+0000 7fb3327fc700 1 --2- 172.21.15.59:0/3103666062 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb3200692a0 0x7fb32006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:01.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.653+0000 7fb3327fc700 1 --2- 172.21.15.59:0/3103666062 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb3200692a0 0x7fb32006b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb31c00b3b0 tx=0x7fb31c009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:01.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.657+0000 7fb32b7fe700 1 -- 172.21.15.59:0/3103666062 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fb324056b10 con 0x7fb334066fd0 2024-03-19T03:14:01.659 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:14:01.659 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:01 smithi059 ceph-mon[29133]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-03-19T03:14:01.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.855+0000 7fb3396fe700 1 -- 172.21.15.59:0/3103666062 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7fb3341058e0 con 0x7fb334066fd0 2024-03-19T03:14:01.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.855+0000 7fb32b7fe700 1 -- 172.21.15.59:0/3103666062 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7fb3240566a0 con 0x7fb334066fd0 2024-03-19T03:14:01.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.857+0000 7fb3396fe700 1 -- 172.21.15.59:0/3103666062 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb3200692a0 msgr2=0x7fb32006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:01.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.857+0000 7fb3396fe700 1 --2- 172.21.15.59:0/3103666062 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb3200692a0 0x7fb32006b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb31c00b3b0 tx=0x7fb31c009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:01.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.857+0000 7fb3396fe700 1 -- 172.21.15.59:0/3103666062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb334066fd0 msgr2=0x7fb3340fd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:01.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.857+0000 7fb3396fe700 1 --2- 172.21.15.59:0/3103666062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb334066fd0 0x7fb3340fd1d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb324002a50 tx=0x7fb3240046b0 comp rx=0 tx=0).stop 2024-03-19T03:14:01.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.858+0000 7fb3396fe700 1 -- 172.21.15.59:0/3103666062 shutdown_connections 2024-03-19T03:14:01.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.858+0000 7fb3396fe700 1 --2- 172.21.15.59:0/3103666062 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb3200692a0 0x7fb32006b750 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:01.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.858+0000 7fb3396fe700 1 --2- 172.21.15.59:0/3103666062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb334066fd0 0x7fb3340fd1d0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:01.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.858+0000 7fb3396fe700 1 --2- 172.21.15.59:0/3103666062 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3340fff10 0x7fb3340fb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:01.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.858+0000 7fb3396fe700 1 -- 172.21.15.59:0/3103666062 >> 172.21.15.59:0/3103666062 conn(0x7fb3340f7440 msgr2=0x7fb3340f7860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:01.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.858+0000 7fb3396fe700 1 -- 172.21.15.59:0/3103666062 shutdown_connections 2024-03-19T03:14:01.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:01.858+0000 7fb3396fe700 1 -- 172.21.15.59:0/3103666062 wait complete. 2024-03-19T03:14:02.540 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -e sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-19T03:14:02.826 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:14:04.321 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:04 smithi059 ceph-mon[29133]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-03-19T03:14:04.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:04 smithi176 ceph-mon[36450]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-03-19T03:14:04.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.700+0000 7f099f980700 1 -- 172.21.15.59:0/3946478305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09980fbcd0 msgr2=0x7f09980fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:04.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.700+0000 7f099f980700 1 --2- 172.21.15.59:0/3946478305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09980fbcd0 0x7f09980fc0e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0994004940 tx=0x7f0994009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:04.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.700+0000 7f099f980700 1 -- 172.21.15.59:0/3946478305 shutdown_connections 2024-03-19T03:14:04.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.700+0000 7f099f980700 1 --2- 172.21.15.59:0/3946478305 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09980fcf80 0x7f09980fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:04.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.700+0000 7f099f980700 1 --2- 172.21.15.59:0/3946478305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09980fbcd0 0x7f09980fc0e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:04.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.700+0000 7f099f980700 1 -- 172.21.15.59:0/3946478305 >> 172.21.15.59:0/3946478305 conn(0x7f09980f7430 msgr2=0x7f09980f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:04.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.700+0000 7f099f980700 1 -- 172.21.15.59:0/3946478305 shutdown_connections 2024-03-19T03:14:04.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.700+0000 7f099f980700 1 -- 172.21.15.59:0/3946478305 wait complete. 2024-03-19T03:14:04.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.701+0000 7f099f980700 1 Processor -- start 2024-03-19T03:14:04.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.701+0000 7f099f980700 1 -- start start 2024-03-19T03:14:04.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.701+0000 7f099f980700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09980fbcd0 0x7f0998110720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:04.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.701+0000 7f099f980700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09980fcf80 0x7f0998112c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:04.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.701+0000 7f099f980700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0998066370 con 0x7f09980fbcd0 2024-03-19T03:14:04.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.701+0000 7f099f980700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09980664b0 con 0x7f09980fcf80 2024-03-19T03:14:04.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.702+0000 7f099cf1b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09980fcf80 0x7f0998112c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:04.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.702+0000 7f099cf1b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09980fcf80 0x7f0998112c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.59:42122/0 (socket says 172.21.15.59:42122) 2024-03-19T03:14:04.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.702+0000 7f099cf1b700 1 -- 172.21.15.59:0/4071621174 learned_addr learned my addr 172.21.15.59:0/4071621174 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:14:04.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.702+0000 7f099d71c700 1 --2- 172.21.15.59:0/4071621174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09980fbcd0 0x7f0998110720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:04.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.702+0000 7f099cf1b700 1 -- 172.21.15.59:0/4071621174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09980fbcd0 msgr2=0x7f0998110720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:04.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.702+0000 7f099cf1b700 1 --2- 172.21.15.59:0/4071621174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09980fbcd0 0x7f0998110720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:04.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.702+0000 7f099cf1b700 1 -- 172.21.15.59:0/4071621174 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0994005040 con 0x7f09980fcf80 2024-03-19T03:14:04.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.702+0000 7f099d71c700 1 --2- 172.21.15.59:0/4071621174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09980fbcd0 0x7f0998110720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:14:04.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.702+0000 7f099cf1b700 1 --2- 172.21.15.59:0/4071621174 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09980fcf80 0x7f0998112c70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0988002a50 tx=0x7f09880046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:04.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.702+0000 7f098dffb700 1 -- 172.21.15.59:0/4071621174 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0988007cc0 con 0x7f09980fcf80 2024-03-19T03:14:04.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.702+0000 7f099f980700 1 -- 172.21.15.59:0/4071621174 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0998113270 con 0x7f09980fcf80 2024-03-19T03:14:04.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.702+0000 7f098dffb700 1 -- 172.21.15.59:0/4071621174 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f098800f040 con 0x7f09980fcf80 2024-03-19T03:14:04.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.702+0000 7f098dffb700 1 -- 172.21.15.59:0/4071621174 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f098800da30 con 0x7f09980fcf80 2024-03-19T03:14:04.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.703+0000 7f099f980700 1 -- 172.21.15.59:0/4071621174 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0998113800 con 0x7f09980fcf80 2024-03-19T03:14:04.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.703+0000 7f099f980700 1 -- 172.21.15.59:0/4071621174 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f099805f5f0 con 0x7f09980fcf80 2024-03-19T03:14:04.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.704+0000 7f098dffb700 1 -- 172.21.15.59:0/4071621174 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f098800db90 con 0x7f09980fcf80 2024-03-19T03:14:04.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.704+0000 7f098dffb700 1 --2- 172.21.15.59:0/4071621174 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f09840692a0 0x7f098406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:04.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.704+0000 7f098dffb700 1 -- 172.21.15.59:0/4071621174 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f0988086f40 con 0x7f09980fcf80 2024-03-19T03:14:04.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.704+0000 7f099d71c700 1 --2- 172.21.15.59:0/4071621174 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f09840692a0 0x7f098406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:04.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.705+0000 7f099d71c700 1 --2- 172.21.15.59:0/4071621174 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f09840692a0 0x7f098406b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f099400b970 tx=0x7f0994009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:04.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.709+0000 7f098dffb700 1 -- 172.21.15.59:0/4071621174 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f09880531f0 con 0x7f09980fcf80 2024-03-19T03:14:04.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.911+0000 7f099f980700 1 -- 172.21.15.59:0/4071621174 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f099804bed0 con 0x7f09980fcf80 2024-03-19T03:14:04.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.913+0000 7f098dffb700 1 -- 172.21.15.59:0/4071621174 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f0988056810 con 0x7f09980fcf80 2024-03-19T03:14:04.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.915+0000 7f099f980700 1 -- 172.21.15.59:0/4071621174 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f09840692a0 msgr2=0x7f098406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:04.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.915+0000 7f099f980700 1 --2- 172.21.15.59:0/4071621174 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f09840692a0 0x7f098406b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f099400b970 tx=0x7f0994009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:04.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.915+0000 7f099f980700 1 -- 172.21.15.59:0/4071621174 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09980fcf80 msgr2=0x7f0998112c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:04.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.915+0000 7f099f980700 1 --2- 172.21.15.59:0/4071621174 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09980fcf80 0x7f0998112c70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0988002a50 tx=0x7f09880046b0 comp rx=0 tx=0).stop 2024-03-19T03:14:04.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.915+0000 7f099f980700 1 -- 172.21.15.59:0/4071621174 shutdown_connections 2024-03-19T03:14:04.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.915+0000 7f099f980700 1 --2- 172.21.15.59:0/4071621174 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f09840692a0 0x7f098406b750 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:04.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.915+0000 7f099f980700 1 --2- 172.21.15.59:0/4071621174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09980fbcd0 0x7f0998110720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:04.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.915+0000 7f099f980700 1 --2- 172.21.15.59:0/4071621174 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09980fcf80 0x7f0998112c70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:04.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.915+0000 7f099f980700 1 -- 172.21.15.59:0/4071621174 >> 172.21.15.59:0/4071621174 conn(0x7f09980f7430 msgr2=0x7f09980f8b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:04.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.915+0000 7f099f980700 1 -- 172.21.15.59:0/4071621174 shutdown_connections 2024-03-19T03:14:04.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:04.915+0000 7f099f980700 1 -- 172.21.15.59:0/4071621174 wait complete. 2024-03-19T03:14:05.707 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -e sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-19T03:14:05.974 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:14:06.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:06 smithi059 ceph-mon[29133]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-03-19T03:14:06.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:06 smithi176 ceph-mon[36450]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-03-19T03:14:07.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.834+0000 7fdb8a67a700 1 -- 172.21.15.59:0/3129041014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb840fc2b0 msgr2=0x7fdb840fc6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:07.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.834+0000 7fdb8a67a700 1 --2- 172.21.15.59:0/3129041014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb840fc2b0 0x7fdb840fc6c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdb74004940 tx=0x7fdb74009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:07.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.834+0000 7fdb8a67a700 1 -- 172.21.15.59:0/3129041014 shutdown_connections 2024-03-19T03:14:07.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.834+0000 7fdb8a67a700 1 --2- 172.21.15.59:0/3129041014 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb840fcc00 0x7fdb840fd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:07.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.834+0000 7fdb8a67a700 1 --2- 172.21.15.59:0/3129041014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb840fc2b0 0x7fdb840fc6c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:07.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.834+0000 7fdb8a67a700 1 -- 172.21.15.59:0/3129041014 >> 172.21.15.59:0/3129041014 conn(0x7fdb840f6ba0 msgr2=0x7fdb840f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:07.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.834+0000 7fdb8a67a700 1 -- 172.21.15.59:0/3129041014 shutdown_connections 2024-03-19T03:14:07.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.834+0000 7fdb8a67a700 1 -- 172.21.15.59:0/3129041014 wait complete. 2024-03-19T03:14:07.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.835+0000 7fdb8a67a700 1 Processor -- start 2024-03-19T03:14:07.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.835+0000 7fdb8a67a700 1 -- start start 2024-03-19T03:14:07.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.835+0000 7fdb8a67a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb840fc2b0 0x7fdb8410da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:07.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.835+0000 7fdb8a67a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb840fcc00 0x7fdb8410df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:07.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.835+0000 7fdb8a67a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8410e590 con 0x7fdb840fcc00 2024-03-19T03:14:07.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.835+0000 7fdb8a67a700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8410ed90 con 0x7fdb840fc2b0 2024-03-19T03:14:07.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.835+0000 7fdb837fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb840fcc00 0x7fdb8410df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:07.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.835+0000 7fdb83fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb840fc2b0 0x7fdb8410da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:07.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.835+0000 7fdb837fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb840fcc00 0x7fdb8410df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41882/0 (socket says 172.21.15.59:41882) 2024-03-19T03:14:07.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.835+0000 7fdb837fe700 1 -- 172.21.15.59:0/2778208614 learned_addr learned my addr 172.21.15.59:0/2778208614 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:14:07.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.836+0000 7fdb837fe700 1 -- 172.21.15.59:0/2778208614 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb840fc2b0 msgr2=0x7fdb8410da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:07.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.836+0000 7fdb837fe700 1 --2- 172.21.15.59:0/2778208614 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb840fc2b0 0x7fdb8410da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:07.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.836+0000 7fdb837fe700 1 -- 172.21.15.59:0/2778208614 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb74005040 con 0x7fdb840fcc00 2024-03-19T03:14:07.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.836+0000 7fdb837fe700 1 --2- 172.21.15.59:0/2778208614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb840fcc00 0x7fdb8410df90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdb78002a80 tx=0x7fdb780046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:07.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.836+0000 7fdb80ff9700 1 -- 172.21.15.59:0/2778208614 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb78007cc0 con 0x7fdb840fcc00 2024-03-19T03:14:07.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.836+0000 7fdb80ff9700 1 -- 172.21.15.59:0/2778208614 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb7800f040 con 0x7fdb840fcc00 2024-03-19T03:14:07.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.836+0000 7fdb8a67a700 1 -- 172.21.15.59:0/2778208614 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb8410f070 con 0x7fdb840fcc00 2024-03-19T03:14:07.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.836+0000 7fdb80ff9700 1 -- 172.21.15.59:0/2778208614 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb7800dac0 con 0x7fdb840fcc00 2024-03-19T03:14:07.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.836+0000 7fdb8a67a700 1 -- 172.21.15.59:0/2778208614 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb8410f5b0 con 0x7fdb840fcc00 2024-03-19T03:14:07.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.837+0000 7fdb8a67a700 1 -- 172.21.15.59:0/2778208614 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb84101c00 con 0x7fdb840fcc00 2024-03-19T03:14:07.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.838+0000 7fdb80ff9700 1 -- 172.21.15.59:0/2778208614 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fdb7800dc20 con 0x7fdb840fcc00 2024-03-19T03:14:07.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.838+0000 7fdb80ff9700 1 --2- 172.21.15.59:0/2778208614 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fdb6c0692a0 0x7fdb6c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:07.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.838+0000 7fdb80ff9700 1 -- 172.21.15.59:0/2778208614 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fdb78086dd0 con 0x7fdb840fcc00 2024-03-19T03:14:07.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.838+0000 7fdb83fff700 1 --2- 172.21.15.59:0/2778208614 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fdb6c0692a0 0x7fdb6c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:07.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.839+0000 7fdb83fff700 1 --2- 172.21.15.59:0/2778208614 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fdb6c0692a0 0x7fdb6c06b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdb7400b970 tx=0x7fdb74009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:07.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:07.843+0000 7fdb80ff9700 1 -- 172.21.15.59:0/2778208614 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fdb78056b10 con 0x7fdb840fcc00 2024-03-19T03:14:08.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:08.041+0000 7fdb8a67a700 1 -- 172.21.15.59:0/2778208614 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7fdb8405f5f0 con 0x7fdb840fcc00 2024-03-19T03:14:08.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:08.042+0000 7fdb80ff9700 1 -- 172.21.15.59:0/2778208614 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7fdb780566a0 con 0x7fdb840fcc00 2024-03-19T03:14:08.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:08.043+0000 7fdb8a67a700 1 -- 172.21.15.59:0/2778208614 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fdb6c0692a0 msgr2=0x7fdb6c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:08.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:08.043+0000 7fdb8a67a700 1 --2- 172.21.15.59:0/2778208614 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fdb6c0692a0 0x7fdb6c06b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdb7400b970 tx=0x7fdb74009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:08.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:08.043+0000 7fdb8a67a700 1 -- 172.21.15.59:0/2778208614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb840fcc00 msgr2=0x7fdb8410df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:08.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:08.043+0000 7fdb8a67a700 1 --2- 172.21.15.59:0/2778208614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb840fcc00 0x7fdb8410df90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdb78002a80 tx=0x7fdb780046b0 comp rx=0 tx=0).stop 2024-03-19T03:14:08.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:08.044+0000 7fdb8a67a700 1 -- 172.21.15.59:0/2778208614 shutdown_connections 2024-03-19T03:14:08.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:08.044+0000 7fdb8a67a700 1 --2- 172.21.15.59:0/2778208614 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fdb6c0692a0 0x7fdb6c06b750 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:08.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:08.044+0000 7fdb8a67a700 1 --2- 172.21.15.59:0/2778208614 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb840fc2b0 0x7fdb8410da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:08.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:08.044+0000 7fdb8a67a700 1 --2- 172.21.15.59:0/2778208614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb840fcc00 0x7fdb8410df90 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:08.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:08.044+0000 7fdb8a67a700 1 -- 172.21.15.59:0/2778208614 >> 172.21.15.59:0/2778208614 conn(0x7fdb840f6ba0 msgr2=0x7fdb840fb520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:08.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:08.044+0000 7fdb8a67a700 1 -- 172.21.15.59:0/2778208614 shutdown_connections 2024-03-19T03:14:08.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:08.044+0000 7fdb8a67a700 1 -- 172.21.15.59:0/2778208614 wait complete. 2024-03-19T03:14:08.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:08 smithi059 ceph-mon[29133]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-19T03:14:08.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:08 smithi176 ceph-mon[36450]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-19T03:14:08.892 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -e sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-19T03:14:09.174 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:14:10.311 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:10 smithi059 ceph-mon[29133]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-19T03:14:10.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:10 smithi176 ceph-mon[36450]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-19T03:14:10.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.931+0000 7f564680f700 1 -- 172.21.15.59:0/4197697818 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56400fbcd0 msgr2=0x7f56400fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:10.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.931+0000 7f564680f700 1 --2- 172.21.15.59:0/4197697818 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56400fbcd0 0x7f56400fc0e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5630004940 tx=0x7f5630009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:10.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.931+0000 7f564680f700 1 -- 172.21.15.59:0/4197697818 shutdown_connections 2024-03-19T03:14:10.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.931+0000 7f564680f700 1 --2- 172.21.15.59:0/4197697818 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56400fcf80 0x7f56400fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:10.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.931+0000 7f564680f700 1 --2- 172.21.15.59:0/4197697818 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56400fbcd0 0x7f56400fc0e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:10.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.931+0000 7f564680f700 1 -- 172.21.15.59:0/4197697818 >> 172.21.15.59:0/4197697818 conn(0x7f56400f7430 msgr2=0x7f56400f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:10.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.931+0000 7f564680f700 1 -- 172.21.15.59:0/4197697818 shutdown_connections 2024-03-19T03:14:10.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.931+0000 7f564680f700 1 -- 172.21.15.59:0/4197697818 wait complete. 2024-03-19T03:14:10.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.931+0000 7f564680f700 1 Processor -- start 2024-03-19T03:14:10.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.932+0000 7f564680f700 1 -- start start 2024-03-19T03:14:10.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.932+0000 7f564680f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56400fbcd0 0x7f564010fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:10.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.932+0000 7f564680f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56400fcf80 0x7f5640110130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:10.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.932+0000 7f564680f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5640110730 con 0x7f56400fbcd0 2024-03-19T03:14:10.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.932+0000 7f564680f700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5640110f30 con 0x7f56400fcf80 2024-03-19T03:14:10.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.932+0000 7f563ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56400fbcd0 0x7f564010fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:10.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.932+0000 7f563ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56400fbcd0 0x7f564010fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41914/0 (socket says 172.21.15.59:41914) 2024-03-19T03:14:10.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.932+0000 7f563ffff700 1 -- 172.21.15.59:0/2318169891 learned_addr learned my addr 172.21.15.59:0/2318169891 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:14:10.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.932+0000 7f563f7fe700 1 --2- 172.21.15.59:0/2318169891 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56400fcf80 0x7f5640110130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:10.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.932+0000 7f563ffff700 1 -- 172.21.15.59:0/2318169891 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56400fcf80 msgr2=0x7f5640110130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:10.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.932+0000 7f563ffff700 1 --2- 172.21.15.59:0/2318169891 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56400fcf80 0x7f5640110130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:10.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.932+0000 7f563ffff700 1 -- 172.21.15.59:0/2318169891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5630005040 con 0x7f56400fbcd0 2024-03-19T03:14:10.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.932+0000 7f563ffff700 1 --2- 172.21.15.59:0/2318169891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56400fbcd0 0x7f564010fbf0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f563000b3b0 tx=0x7f5630009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:10.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.933+0000 7f563cff9700 1 -- 172.21.15.59:0/2318169891 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f563000e040 con 0x7f56400fbcd0 2024-03-19T03:14:10.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.933+0000 7f564680f700 1 -- 172.21.15.59:0/2318169891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56401111b0 con 0x7f56400fbcd0 2024-03-19T03:14:10.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.933+0000 7f563cff9700 1 -- 172.21.15.59:0/2318169891 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f563000bbf0 con 0x7f56400fbcd0 2024-03-19T03:14:10.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.933+0000 7f563cff9700 1 -- 172.21.15.59:0/2318169891 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f563001c630 con 0x7f56400fbcd0 2024-03-19T03:14:10.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.933+0000 7f564680f700 1 -- 172.21.15.59:0/2318169891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56401116f0 con 0x7f56400fbcd0 2024-03-19T03:14:10.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.934+0000 7f564680f700 1 -- 172.21.15.59:0/2318169891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5640101910 con 0x7f56400fbcd0 2024-03-19T03:14:10.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.934+0000 7f563cff9700 1 -- 172.21.15.59:0/2318169891 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f563000bd60 con 0x7f56400fbcd0 2024-03-19T03:14:10.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.935+0000 7f563cff9700 1 --2- 172.21.15.59:0/2318169891 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f56280692a0 0x7f562806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:10.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.935+0000 7f563cff9700 1 -- 172.21.15.59:0/2318169891 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f5630087e40 con 0x7f56400fbcd0 2024-03-19T03:14:10.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.935+0000 7f563f7fe700 1 --2- 172.21.15.59:0/2318169891 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f56280692a0 0x7f562806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:10.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.935+0000 7f563f7fe700 1 --2- 172.21.15.59:0/2318169891 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f56280692a0 0x7f562806b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5634009fd0 tx=0x7f563400b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:10.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:10.942+0000 7f563cff9700 1 -- 172.21.15.59:0/2318169891 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f5630057c00 con 0x7f56400fbcd0 2024-03-19T03:14:11.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:11.173+0000 7f564680f700 1 -- 172.21.15.59:0/2318169891 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810", "target": ["mon-mgr", ""]}) v1 -- 0x7f5640111910 con 0x7f56280692a0 2024-03-19T03:14:11.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:11.179+0000 7f563cff9700 1 -- 172.21.15.59:0/2318169891 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f5640111910 con 0x7f56280692a0 2024-03-19T03:14:11.180 INFO:teuthology.orchestra.run.smithi059.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T03:14:11.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:11.181+0000 7f564680f700 1 -- 172.21.15.59:0/2318169891 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f56280692a0 msgr2=0x7f562806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:11.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:11.181+0000 7f564680f700 1 --2- 172.21.15.59:0/2318169891 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f56280692a0 0x7f562806b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5634009fd0 tx=0x7f563400b040 comp rx=0 tx=0).stop 2024-03-19T03:14:11.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:11.181+0000 7f564680f700 1 -- 172.21.15.59:0/2318169891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56400fbcd0 msgr2=0x7f564010fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:11.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:11.181+0000 7f564680f700 1 --2- 172.21.15.59:0/2318169891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56400fbcd0 0x7f564010fbf0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f563000b3b0 tx=0x7f5630009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:11.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:11.181+0000 7f564680f700 1 -- 172.21.15.59:0/2318169891 shutdown_connections 2024-03-19T03:14:11.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:11.181+0000 7f564680f700 1 --2- 172.21.15.59:0/2318169891 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f56280692a0 0x7f562806b750 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:11.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:11.181+0000 7f564680f700 1 --2- 172.21.15.59:0/2318169891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56400fbcd0 0x7f564010fbf0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:11.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:11.181+0000 7f564680f700 1 --2- 172.21.15.59:0/2318169891 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56400fcf80 0x7f5640110130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:11.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:11.181+0000 7f564680f700 1 -- 172.21.15.59:0/2318169891 >> 172.21.15.59:0/2318169891 conn(0x7f56400f7430 msgr2=0x7f5640065b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:11.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:11.181+0000 7f564680f700 1 -- 172.21.15.59:0/2318169891 shutdown_connections 2024-03-19T03:14:11.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:11.181+0000 7f564680f700 1 -- 172.21.15.59:0/2318169891 wait complete. 2024-03-19T03:14:11.990 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-19T03:14:11.991 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2024-03-19T03:14:11.991 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -e sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-03-19T03:14:12.296 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:14:12.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:12 smithi059 ceph-mon[29133]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-19T03:14:12.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:12 smithi059 ceph-mon[29133]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:14:12.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:12 smithi059 ceph-mon[29133]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T03:14:12.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:12 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:14:12.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:12 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:14:12.627 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:12 smithi176 ceph-mon[36450]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-19T03:14:12.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:12 smithi176 ceph-mon[36450]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:14:12.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:12 smithi176 ceph-mon[36450]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T03:14:12.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:12 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:14:12.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:12 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:14:13.445 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:13 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:14:13.446 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:13 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:14:13.446 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:13 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:14:13.627 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:13 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:14:13.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:13 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:14:13.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:13 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:14:14.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.184+0000 7f79cfc34700 1 -- 172.21.15.59:0/1007985541 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79c80fe440 msgr2=0x7f79c80fe850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:14.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.184+0000 7f79cfc34700 1 --2- 172.21.15.59:0/1007985541 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79c80fe440 0x7f79c80fe850 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f79c4004940 tx=0x7f79c4009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:14.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.184+0000 7f79cfc34700 1 -- 172.21.15.59:0/1007985541 shutdown_connections 2024-03-19T03:14:14.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.184+0000 7f79cfc34700 1 --2- 172.21.15.59:0/1007985541 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79c80fed90 0x7f79c80faed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:14.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.184+0000 7f79cfc34700 1 --2- 172.21.15.59:0/1007985541 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79c80fe440 0x7f79c80fe850 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:14.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.184+0000 7f79cfc34700 1 -- 172.21.15.59:0/1007985541 >> 172.21.15.59:0/1007985541 conn(0x7f79c80f6be0 msgr2=0x7f79c80f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:14.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.184+0000 7f79cfc34700 1 -- 172.21.15.59:0/1007985541 shutdown_connections 2024-03-19T03:14:14.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.184+0000 7f79cfc34700 1 -- 172.21.15.59:0/1007985541 wait complete. 2024-03-19T03:14:14.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.185+0000 7f79cfc34700 1 Processor -- start 2024-03-19T03:14:14.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.185+0000 7f79cfc34700 1 -- start start 2024-03-19T03:14:14.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.185+0000 7f79cfc34700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79c80fe440 0x7f79c810d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:14.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.185+0000 7f79cfc34700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79c80fed90 0x7f79c810de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:14.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.185+0000 7f79cfc34700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c810e410 con 0x7f79c80fed90 2024-03-19T03:14:14.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.185+0000 7f79cfc34700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c810e580 con 0x7f79c80fe440 2024-03-19T03:14:14.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.185+0000 7f79cd9d0700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79c80fe440 0x7f79c810d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:14.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.185+0000 7f79cd9d0700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79c80fe440 0x7f79c810d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.59:36600/0 (socket says 172.21.15.59:36600) 2024-03-19T03:14:14.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.185+0000 7f79cd9d0700 1 -- 172.21.15.59:0/1887748839 learned_addr learned my addr 172.21.15.59:0/1887748839 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:14:14.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.185+0000 7f79cd1cf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79c80fed90 0x7f79c810de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:14.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.185+0000 7f79cd9d0700 1 -- 172.21.15.59:0/1887748839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79c80fed90 msgr2=0x7f79c810de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:14.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.185+0000 7f79cd9d0700 1 --2- 172.21.15.59:0/1887748839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79c80fed90 0x7f79c810de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:14.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.185+0000 7f79cd9d0700 1 -- 172.21.15.59:0/1887748839 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c4005040 con 0x7f79c80fe440 2024-03-19T03:14:14.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.186+0000 7f79cd9d0700 1 --2- 172.21.15.59:0/1887748839 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79c80fe440 0x7f79c810d8d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f79c4013870 tx=0x7f79c4009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:14.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.186+0000 7f79be7fc700 1 -- 172.21.15.59:0/1887748839 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79c400e070 con 0x7f79c80fe440 2024-03-19T03:14:14.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.186+0000 7f79cfc34700 1 -- 172.21.15.59:0/1887748839 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79c80fbfe0 con 0x7f79c80fe440 2024-03-19T03:14:14.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.186+0000 7f79cfc34700 1 -- 172.21.15.59:0/1887748839 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79c80fc4c0 con 0x7f79c80fe440 2024-03-19T03:14:14.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.186+0000 7f79be7fc700 1 -- 172.21.15.59:0/1887748839 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f79c4013de0 con 0x7f79c80fe440 2024-03-19T03:14:14.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.186+0000 7f79be7fc700 1 -- 172.21.15.59:0/1887748839 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79c401ce70 con 0x7f79c80fe440 2024-03-19T03:14:14.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.187+0000 7f79cfc34700 1 -- 172.21.15.59:0/1887748839 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79ac0052e0 con 0x7f79c80fe440 2024-03-19T03:14:14.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.187+0000 7f79be7fc700 1 -- 172.21.15.59:0/1887748839 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f79c4013910 con 0x7f79c80fe440 2024-03-19T03:14:14.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.188+0000 7f79be7fc700 1 --2- 172.21.15.59:0/1887748839 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f79b40692a0 0x7f79b406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:14.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.188+0000 7f79be7fc700 1 -- 172.21.15.59:0/1887748839 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f79c40887e0 con 0x7f79c80fe440 2024-03-19T03:14:14.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.188+0000 7f79cd1cf700 1 --2- 172.21.15.59:0/1887748839 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f79b40692a0 0x7f79b406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:14.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.189+0000 7f79cd1cf700 1 --2- 172.21.15.59:0/1887748839 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f79b40692a0 0x7f79b406b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f79b8009b80 tx=0x7f79b8014040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:14.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.193+0000 7f79be7fc700 1 -- 172.21.15.59:0/1887748839 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f79c4058500 con 0x7f79c80fe440 2024-03-19T03:14:14.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:14 smithi059 ceph-mon[29133]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T03:14:14.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:14 smithi059 ceph-mon[29133]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-19T03:14:14.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.542+0000 7f79cfc34700 1 -- 172.21.15.59:0/1887748839 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f79ac000bf0 con 0x7f79b40692a0 2024-03-19T03:14:14.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.543+0000 7f79be7fc700 1 -- 172.21.15.59:0/1887748839 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f79ac000bf0 con 0x7f79b40692a0 2024-03-19T03:14:14.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.545+0000 7f79cfc34700 1 -- 172.21.15.59:0/1887748839 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f79b40692a0 msgr2=0x7f79b406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:14.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.545+0000 7f79cfc34700 1 --2- 172.21.15.59:0/1887748839 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f79b40692a0 0x7f79b406b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f79b8009b80 tx=0x7f79b8014040 comp rx=0 tx=0).stop 2024-03-19T03:14:14.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.545+0000 7f79cfc34700 1 -- 172.21.15.59:0/1887748839 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79c80fe440 msgr2=0x7f79c810d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:14.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.545+0000 7f79cfc34700 1 --2- 172.21.15.59:0/1887748839 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79c80fe440 0x7f79c810d8d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f79c4013870 tx=0x7f79c4009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:14.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.545+0000 7f79cfc34700 1 -- 172.21.15.59:0/1887748839 shutdown_connections 2024-03-19T03:14:14.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.545+0000 7f79cfc34700 1 --2- 172.21.15.59:0/1887748839 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f79b40692a0 0x7f79b406b750 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:14.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.545+0000 7f79cfc34700 1 --2- 172.21.15.59:0/1887748839 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79c80fe440 0x7f79c810d8d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:14.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.545+0000 7f79cfc34700 1 --2- 172.21.15.59:0/1887748839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79c80fed90 0x7f79c810de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:14.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.545+0000 7f79cfc34700 1 -- 172.21.15.59:0/1887748839 >> 172.21.15.59:0/1887748839 conn(0x7f79c80f6be0 msgr2=0x7f79c80f8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:14.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.545+0000 7f79cfc34700 1 -- 172.21.15.59:0/1887748839 shutdown_connections 2024-03-19T03:14:14.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.545+0000 7f79cfc34700 1 -- 172.21.15.59:0/1887748839 wait complete. 2024-03-19T03:14:14.558 INFO:teuthology.orchestra.run.smithi059.stdout:true 2024-03-19T03:14:14.627 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:14 smithi176 ceph-mon[36450]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T03:14:14.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:14 smithi176 ceph-mon[36450]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-19T03:14:14.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.677+0000 7f7cfb59e700 1 -- 172.21.15.59:0/2396500439 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf40fda50 msgr2=0x7f7cf40fbe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:14.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.677+0000 7f7cfb59e700 1 --2- 172.21.15.59:0/2396500439 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf40fda50 0x7f7cf40fbe60 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7cec004940 tx=0x7f7cec009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:14.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.677+0000 7f7cfb59e700 1 -- 172.21.15.59:0/2396500439 shutdown_connections 2024-03-19T03:14:14.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.677+0000 7f7cfb59e700 1 --2- 172.21.15.59:0/2396500439 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf40fda50 0x7f7cf40fbe60 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:14.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.677+0000 7f7cfb59e700 1 --2- 172.21.15.59:0/2396500439 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cf40fd100 0x7f7cf40fd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:14.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.677+0000 7f7cfb59e700 1 -- 172.21.15.59:0/2396500439 >> 172.21.15.59:0/2396500439 conn(0x7f7cf40f6a00 msgr2=0x7f7cf40f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:14.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.678+0000 7f7cfb59e700 1 -- 172.21.15.59:0/2396500439 shutdown_connections 2024-03-19T03:14:14.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.678+0000 7f7cfb59e700 1 -- 172.21.15.59:0/2396500439 wait complete. 2024-03-19T03:14:14.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.678+0000 7f7cfb59e700 1 Processor -- start 2024-03-19T03:14:14.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.678+0000 7f7cfb59e700 1 -- start start 2024-03-19T03:14:14.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.679+0000 7f7cfb59e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf40fd100 0x7f7cf4110500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:14.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.679+0000 7f7cfb59e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cf40fda50 0x7f7cf4112a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:14.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.679+0000 7f7cfb59e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf40fbac0 con 0x7f7cf40fd100 2024-03-19T03:14:14.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.679+0000 7f7cfb59e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf40fbc00 con 0x7f7cf40fda50 2024-03-19T03:14:14.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.679+0000 7f7cfa59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf40fd100 0x7f7cf4110500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:14.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.679+0000 7f7cfa59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf40fd100 0x7f7cf4110500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52426/0 (socket says 172.21.15.59:52426) 2024-03-19T03:14:14.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.679+0000 7f7cfa59c700 1 -- 172.21.15.59:0/1803720759 learned_addr learned my addr 172.21.15.59:0/1803720759 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:14:14.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.679+0000 7f7cf9d9b700 1 --2- 172.21.15.59:0/1803720759 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cf40fda50 0x7f7cf4112a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:14.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.679+0000 7f7cfa59c700 1 -- 172.21.15.59:0/1803720759 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cf40fda50 msgr2=0x7f7cf4112a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:14.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.679+0000 7f7cfa59c700 1 --2- 172.21.15.59:0/1803720759 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cf40fda50 0x7f7cf4112a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:14.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.679+0000 7f7cfa59c700 1 -- 172.21.15.59:0/1803720759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cec005040 con 0x7f7cf40fd100 2024-03-19T03:14:14.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.679+0000 7f7cfa59c700 1 --2- 172.21.15.59:0/1803720759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf40fd100 0x7f7cf4110500 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7ce800eb40 tx=0x7f7ce800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:14.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.679+0000 7f7ce6ffd700 1 -- 172.21.15.59:0/1803720759 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ce8007cc0 con 0x7f7cf40fd100 2024-03-19T03:14:14.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.679+0000 7f7ce6ffd700 1 -- 172.21.15.59:0/1803720759 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ce800f040 con 0x7f7cf40fd100 2024-03-19T03:14:14.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.679+0000 7f7cfb59e700 1 -- 172.21.15.59:0/1803720759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf4112ff0 con 0x7f7cf40fd100 2024-03-19T03:14:14.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.680+0000 7f7ce6ffd700 1 -- 172.21.15.59:0/1803720759 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ce800ca10 con 0x7f7cf40fd100 2024-03-19T03:14:14.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.680+0000 7f7cfb59e700 1 -- 172.21.15.59:0/1803720759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf4113520 con 0x7f7cf40fd100 2024-03-19T03:14:14.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.681+0000 7f7cfb59e700 1 -- 172.21.15.59:0/1803720759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf405f5f0 con 0x7f7cf40fd100 2024-03-19T03:14:14.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.681+0000 7f7ce6ffd700 1 -- 172.21.15.59:0/1803720759 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7ce800cc80 con 0x7f7cf40fd100 2024-03-19T03:14:14.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.682+0000 7f7ce6ffd700 1 --2- 172.21.15.59:0/1803720759 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7ce00692a0 0x7f7ce006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:14.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.682+0000 7f7ce6ffd700 1 -- 172.21.15.59:0/1803720759 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f7ce8086d40 con 0x7f7cf40fd100 2024-03-19T03:14:14.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.682+0000 7f7cf9d9b700 1 --2- 172.21.15.59:0/1803720759 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7ce00692a0 0x7f7ce006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:14.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.682+0000 7f7cf9d9b700 1 --2- 172.21.15.59:0/1803720759 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7ce00692a0 0x7f7ce006b750 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7cec013870 tx=0x7f7cec009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:14.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.687+0000 7f7ce6ffd700 1 -- 172.21.15.59:0/1803720759 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f7ce804de50 con 0x7f7cf40fd100 2024-03-19T03:14:14.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.951+0000 7f7cfb59e700 1 -- 172.21.15.59:0/1803720759 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7cf4101140 con 0x7f7ce00692a0 2024-03-19T03:14:14.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.952+0000 7f7ce6ffd700 1 -- 172.21.15.59:0/1803720759 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f7cf4101140 con 0x7f7ce00692a0 2024-03-19T03:14:14.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.955+0000 7f7cfb59e700 1 -- 172.21.15.59:0/1803720759 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7ce00692a0 msgr2=0x7f7ce006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:14.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.955+0000 7f7cfb59e700 1 --2- 172.21.15.59:0/1803720759 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7ce00692a0 0x7f7ce006b750 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7cec013870 tx=0x7f7cec009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:14.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.955+0000 7f7cfb59e700 1 -- 172.21.15.59:0/1803720759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf40fd100 msgr2=0x7f7cf4110500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:14.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.955+0000 7f7cfb59e700 1 --2- 172.21.15.59:0/1803720759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf40fd100 0x7f7cf4110500 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7ce800eb40 tx=0x7f7ce800b040 comp rx=0 tx=0).stop 2024-03-19T03:14:14.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.955+0000 7f7cfb59e700 1 -- 172.21.15.59:0/1803720759 shutdown_connections 2024-03-19T03:14:14.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.955+0000 7f7cfb59e700 1 --2- 172.21.15.59:0/1803720759 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f7ce00692a0 0x7f7ce006b750 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:14.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.955+0000 7f7cfb59e700 1 --2- 172.21.15.59:0/1803720759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf40fd100 0x7f7cf4110500 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:14.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.955+0000 7f7cfb59e700 1 --2- 172.21.15.59:0/1803720759 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cf40fda50 0x7f7cf4112a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:14.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.955+0000 7f7cfb59e700 1 -- 172.21.15.59:0/1803720759 >> 172.21.15.59:0/1803720759 conn(0x7f7cf40f6a00 msgr2=0x7f7cf40f80a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:14.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.955+0000 7f7cfb59e700 1 -- 172.21.15.59:0/1803720759 shutdown_connections 2024-03-19T03:14:14.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:14.955+0000 7f7cfb59e700 1 -- 172.21.15.59:0/1803720759 wait complete. 2024-03-19T03:14:15.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.084+0000 7fa416874700 1 -- 172.21.15.59:0/985334208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4100fbcd0 msgr2=0x7fa4100fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:15.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.084+0000 7fa416874700 1 --2- 172.21.15.59:0/985334208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4100fbcd0 0x7fa4100fc0e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa400004940 tx=0x7fa400009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:15.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.085+0000 7fa416874700 1 -- 172.21.15.59:0/985334208 shutdown_connections 2024-03-19T03:14:15.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.085+0000 7fa416874700 1 --2- 172.21.15.59:0/985334208 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4100fcf80 0x7fa4100fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:15.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.085+0000 7fa416874700 1 --2- 172.21.15.59:0/985334208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4100fbcd0 0x7fa4100fc0e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:15.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.085+0000 7fa416874700 1 -- 172.21.15.59:0/985334208 >> 172.21.15.59:0/985334208 conn(0x7fa4100f7430 msgr2=0x7fa4100f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:15.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.085+0000 7fa416874700 1 -- 172.21.15.59:0/985334208 shutdown_connections 2024-03-19T03:14:15.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.085+0000 7fa416874700 1 -- 172.21.15.59:0/985334208 wait complete. 2024-03-19T03:14:15.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.085+0000 7fa416874700 1 Processor -- start 2024-03-19T03:14:15.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.085+0000 7fa416874700 1 -- start start 2024-03-19T03:14:15.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.086+0000 7fa416874700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4100fbcd0 0x7fa41010d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:15.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.086+0000 7fa416874700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4100fcf80 0x7fa41010df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:15.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.086+0000 7fa416874700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41010e520 con 0x7fa4100fbcd0 2024-03-19T03:14:15.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.086+0000 7fa416874700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41010ed20 con 0x7fa4100fcf80 2024-03-19T03:14:15.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.086+0000 7fa40ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4100fbcd0 0x7fa41010d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:15.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.086+0000 7fa40ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4100fbcd0 0x7fa41010d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52448/0 (socket says 172.21.15.59:52448) 2024-03-19T03:14:15.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.086+0000 7fa40ffff700 1 -- 172.21.15.59:0/1843256629 learned_addr learned my addr 172.21.15.59:0/1843256629 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:14:15.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.086+0000 7fa40f7fe700 1 --2- 172.21.15.59:0/1843256629 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4100fcf80 0x7fa41010df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:15.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.086+0000 7fa40ffff700 1 -- 172.21.15.59:0/1843256629 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4100fcf80 msgr2=0x7fa41010df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:15.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.086+0000 7fa40ffff700 1 --2- 172.21.15.59:0/1843256629 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4100fcf80 0x7fa41010df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:15.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.086+0000 7fa40ffff700 1 -- 172.21.15.59:0/1843256629 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa400005040 con 0x7fa4100fbcd0 2024-03-19T03:14:15.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.086+0000 7fa40f7fe700 1 --2- 172.21.15.59:0/1843256629 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4100fcf80 0x7fa41010df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:14:15.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.086+0000 7fa40ffff700 1 --2- 172.21.15.59:0/1843256629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4100fbcd0 0x7fa41010d9e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa400013870 tx=0x7fa400009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:15.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.086+0000 7fa40cff9700 1 -- 172.21.15.59:0/1843256629 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa400021070 con 0x7fa4100fbcd0 2024-03-19T03:14:15.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.087+0000 7fa40cff9700 1 -- 172.21.15.59:0/1843256629 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa400009430 con 0x7fa4100fbcd0 2024-03-19T03:14:15.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.087+0000 7fa416874700 1 -- 172.21.15.59:0/1843256629 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa41010efa0 con 0x7fa4100fbcd0 2024-03-19T03:14:15.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.087+0000 7fa416874700 1 -- 172.21.15.59:0/1843256629 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa41010f4e0 con 0x7fa4100fbcd0 2024-03-19T03:14:15.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.087+0000 7fa40cff9700 1 -- 172.21.15.59:0/1843256629 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa40000e070 con 0x7fa4100fbcd0 2024-03-19T03:14:15.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.088+0000 7fa416874700 1 -- 172.21.15.59:0/1843256629 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa410101b10 con 0x7fa4100fbcd0 2024-03-19T03:14:15.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.088+0000 7fa40cff9700 1 -- 172.21.15.59:0/1843256629 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa40001c750 con 0x7fa4100fbcd0 2024-03-19T03:14:15.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.089+0000 7fa40cff9700 1 --2- 172.21.15.59:0/1843256629 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fa3f80692a0 0x7fa3f806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:15.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.089+0000 7fa40cff9700 1 -- 172.21.15.59:0/1843256629 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fa400088360 con 0x7fa4100fbcd0 2024-03-19T03:14:15.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.089+0000 7fa40f7fe700 1 --2- 172.21.15.59:0/1843256629 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fa3f80692a0 0x7fa3f806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:15.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.090+0000 7fa40f7fe700 1 --2- 172.21.15.59:0/1843256629 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fa3f80692a0 0x7fa3f806b750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa404004fd0 tx=0x7fa40400b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:15.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.094+0000 7fa40cff9700 1 -- 172.21.15.59:0/1843256629 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fa4000581c0 con 0x7fa4100fbcd0 2024-03-19T03:14:15.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.349+0000 7fa416874700 1 -- 172.21.15.59:0/1843256629 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4101048f0 con 0x7fa3f80692a0 2024-03-19T03:14:15.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.364+0000 7fa40cff9700 1 -- 172.21.15.59:0/1843256629 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7fa4101048f0 con 0x7fa3f80692a0 2024-03-19T03:14:15.365 INFO:teuthology.orchestra.run.smithi059.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-19T03:14:15.365 INFO:teuthology.orchestra.run.smithi059.stdout:alertmanager.smithi059 smithi059 *:9093,9094 running (4m) 30s ago 6m 27.3M - 0.25.0 c8568f914cd2 40a81ac05a90 2024-03-19T03:14:15.366 INFO:teuthology.orchestra.run.smithi059.stdout:crash.smithi059 smithi059 running (6m) 30s ago 6m 6669k - 17.2.7-700-gb4843bd7 810955603526 3090e46c4bff 2024-03-19T03:14:15.366 INFO:teuthology.orchestra.run.smithi059.stdout:crash.smithi176 smithi176 running (5m) 36s ago 5m 6660k - 17.2.7-700-gb4843bd7 810955603526 2df9a0e61e4c 2024-03-19T03:14:15.366 INFO:teuthology.orchestra.run.smithi059.stdout:grafana.smithi059 smithi059 *:3000 running (4m) 30s ago 6m 85.4M - 9.4.7 954c08fa6188 b3394e3d4026 2024-03-19T03:14:15.366 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi059.doeuxg smithi059 running (49s) 30s ago 48s 14.5M - 17.2.7-700-gb4843bd7 810955603526 18d717af1972 2024-03-19T03:14:15.366 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi059.hmylwa smithi059 running (55s) 30s ago 54s 14.6M - 17.2.7-700-gb4843bd7 810955603526 d4acd92451b4 2024-03-19T03:14:15.366 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi176.bochhq smithi176 running (52s) 36s ago 51s 14.3M - 17.2.7-700-gb4843bd7 810955603526 05e9bc441018 2024-03-19T03:14:15.366 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi176.gisnqc smithi176 running (46s) 36s ago 46s 12.2M - 17.2.7-700-gb4843bd7 810955603526 75982373b447 2024-03-19T03:14:15.366 INFO:teuthology.orchestra.run.smithi059.stdout:mgr.smithi059.azpecc smithi059 *:9283 running (7m) 30s ago 7m 480M - 17.2.7-700-gb4843bd7 810955603526 441ca0cfcfac 2024-03-19T03:14:15.366 INFO:teuthology.orchestra.run.smithi059.stdout:mgr.smithi176.ycmqbw smithi176 *:8443,9283 running (5m) 36s ago 5m 433M - 17.2.7-700-gb4843bd7 810955603526 ac8bc9c6ec9b 2024-03-19T03:14:15.366 INFO:teuthology.orchestra.run.smithi059.stdout:mon.smithi059 smithi059 running (7m) 30s ago 8m 49.9M 2048M 17.2.7-700-gb4843bd7 810955603526 c8bc018eaa37 2024-03-19T03:14:15.367 INFO:teuthology.orchestra.run.smithi059.stdout:mon.smithi176 smithi176 running (5m) 36s ago 5m 33.4M 2048M 17.2.7-700-gb4843bd7 810955603526 4130cce6243b 2024-03-19T03:14:15.367 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter.smithi059 smithi059 *:9100 running (6m) 30s ago 6m 16.1M - 1.5.0 0da6a335fe13 d3fca5e9a667 2024-03-19T03:14:15.367 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter.smithi176 smithi176 *:9100 running (5m) 36s ago 5m 14.7M - 1.5.0 0da6a335fe13 35389a3c33fd 2024-03-19T03:14:15.367 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 smithi059 running (4m) 30s ago 4m 46.3M 4289M 17.2.7-700-gb4843bd7 810955603526 2511279e8852 2024-03-19T03:14:15.367 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 smithi059 running (4m) 30s ago 4m 45.0M 4289M 17.2.7-700-gb4843bd7 810955603526 cd85f348d650 2024-03-19T03:14:15.367 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 smithi059 running (3m) 30s ago 3m 47.0M 4289M 17.2.7-700-gb4843bd7 810955603526 363582327fde 2024-03-19T03:14:15.367 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 smithi176 running (3m) 36s ago 3m 43.5M 5312M 17.2.7-700-gb4843bd7 810955603526 436c587b5265 2024-03-19T03:14:15.367 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 smithi176 running (2m) 36s ago 2m 43.1M 5312M 17.2.7-700-gb4843bd7 810955603526 80d2dd2cd18b 2024-03-19T03:14:15.367 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 smithi176 running (2m) 36s ago 2m 41.6M 5312M 17.2.7-700-gb4843bd7 810955603526 dd0f98b5ec05 2024-03-19T03:14:15.367 INFO:teuthology.orchestra.run.smithi059.stdout:prometheus.smithi059 smithi059 *:9095 running (4m) 30s ago 5m 54.1M - 2.43.0 a07b618ecd1d 1782702019d3 2024-03-19T03:14:15.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.366+0000 7fa416874700 1 -- 172.21.15.59:0/1843256629 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fa3f80692a0 msgr2=0x7fa3f806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:15.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.366+0000 7fa416874700 1 --2- 172.21.15.59:0/1843256629 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fa3f80692a0 0x7fa3f806b750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa404004fd0 tx=0x7fa40400b040 comp rx=0 tx=0).stop 2024-03-19T03:14:15.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.366+0000 7fa416874700 1 -- 172.21.15.59:0/1843256629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4100fbcd0 msgr2=0x7fa41010d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:15.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.366+0000 7fa416874700 1 --2- 172.21.15.59:0/1843256629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4100fbcd0 0x7fa41010d9e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa400013870 tx=0x7fa400009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:15.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.366+0000 7fa416874700 1 -- 172.21.15.59:0/1843256629 shutdown_connections 2024-03-19T03:14:15.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.366+0000 7fa416874700 1 --2- 172.21.15.59:0/1843256629 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fa3f80692a0 0x7fa3f806b750 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:15.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.366+0000 7fa416874700 1 --2- 172.21.15.59:0/1843256629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4100fbcd0 0x7fa41010d9e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:15.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.366+0000 7fa416874700 1 --2- 172.21.15.59:0/1843256629 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4100fcf80 0x7fa41010df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:15.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.366+0000 7fa416874700 1 -- 172.21.15.59:0/1843256629 >> 172.21.15.59:0/1843256629 conn(0x7fa4100f7430 msgr2=0x7fa410065b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:15.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.366+0000 7fa416874700 1 -- 172.21.15.59:0/1843256629 shutdown_connections 2024-03-19T03:14:15.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.366+0000 7fa416874700 1 -- 172.21.15.59:0/1843256629 wait complete. 2024-03-19T03:14:15.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.493+0000 7fd58449c700 1 -- 172.21.15.59:0/2337626530 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd57c0fe070 msgr2=0x7fd57c0fe4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:15.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.493+0000 7fd58449c700 1 --2- 172.21.15.59:0/2337626530 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd57c0fe070 0x7fd57c0fe4e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd56c004940 tx=0x7fd56c009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:15.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.493+0000 7fd58449c700 1 -- 172.21.15.59:0/2337626530 shutdown_connections 2024-03-19T03:14:15.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.493+0000 7fd58449c700 1 --2- 172.21.15.59:0/2337626530 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd57c0fe070 0x7fd57c0fe4e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:15.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.493+0000 7fd58449c700 1 --2- 172.21.15.59:0/2337626530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd57c0fcf00 0x7fd57c0fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:15.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.493+0000 7fd58449c700 1 -- 172.21.15.59:0/2337626530 >> 172.21.15.59:0/2337626530 conn(0x7fd57c0f85a0 msgr2=0x7fd57c0fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:15.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.494+0000 7fd58449c700 1 -- 172.21.15.59:0/2337626530 shutdown_connections 2024-03-19T03:14:15.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.494+0000 7fd58449c700 1 -- 172.21.15.59:0/2337626530 wait complete. 2024-03-19T03:14:15.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.494+0000 7fd58449c700 1 Processor -- start 2024-03-19T03:14:15.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.494+0000 7fd58449c700 1 -- start start 2024-03-19T03:14:15.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.495+0000 7fd58449c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd57c0fcf00 0x7fd57c10d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:15.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.495+0000 7fd58449c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd57c0fe070 0x7fd57c10df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:15.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.495+0000 7fd58449c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c10e510 con 0x7fd57c0fe070 2024-03-19T03:14:15.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.495+0000 7fd58449c700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c10ed60 con 0x7fd57c0fcf00 2024-03-19T03:14:15.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.495+0000 7fd581a37700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd57c0fe070 0x7fd57c10df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:15.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.495+0000 7fd581a37700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd57c0fe070 0x7fd57c10df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52458/0 (socket says 172.21.15.59:52458) 2024-03-19T03:14:15.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.495+0000 7fd581a37700 1 -- 172.21.15.59:0/2229576965 learned_addr learned my addr 172.21.15.59:0/2229576965 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:14:15.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.495+0000 7fd582238700 1 --2- 172.21.15.59:0/2229576965 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd57c0fcf00 0x7fd57c10d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:15.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.495+0000 7fd581a37700 1 -- 172.21.15.59:0/2229576965 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd57c0fcf00 msgr2=0x7fd57c10d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:15.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.495+0000 7fd581a37700 1 --2- 172.21.15.59:0/2229576965 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd57c0fcf00 0x7fd57c10d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:15.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.495+0000 7fd581a37700 1 -- 172.21.15.59:0/2229576965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56c005040 con 0x7fd57c0fe070 2024-03-19T03:14:15.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.495+0000 7fd582238700 1 --2- 172.21.15.59:0/2229576965 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd57c0fcf00 0x7fd57c10d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:14:15.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.495+0000 7fd581a37700 1 --2- 172.21.15.59:0/2229576965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd57c0fe070 0x7fd57c10df10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd56c013870 tx=0x7fd56c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:15.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.495+0000 7fd572ffd700 1 -- 172.21.15.59:0/2229576965 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd56c021070 con 0x7fd57c0fe070 2024-03-19T03:14:15.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.496+0000 7fd572ffd700 1 -- 172.21.15.59:0/2229576965 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd56c01c400 con 0x7fd57c0fe070 2024-03-19T03:14:15.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.496+0000 7fd572ffd700 1 -- 172.21.15.59:0/2229576965 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd56c00e070 con 0x7fd57c0fe070 2024-03-19T03:14:15.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.496+0000 7fd58449c700 1 -- 172.21.15.59:0/2229576965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd57c10efe0 con 0x7fd57c0fe070 2024-03-19T03:14:15.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.496+0000 7fd58449c700 1 -- 172.21.15.59:0/2229576965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd57c10f520 con 0x7fd57c0fe070 2024-03-19T03:14:15.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.497+0000 7fd58449c700 1 -- 172.21.15.59:0/2229576965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd57c05f5f0 con 0x7fd57c0fe070 2024-03-19T03:14:15.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.497+0000 7fd572ffd700 1 -- 172.21.15.59:0/2229576965 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd56c01ca20 con 0x7fd57c0fe070 2024-03-19T03:14:15.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.498+0000 7fd572ffd700 1 --2- 172.21.15.59:0/2229576965 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd5680692a0 0x7fd56806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:15.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.498+0000 7fd572ffd700 1 -- 172.21.15.59:0/2229576965 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fd56c088570 con 0x7fd57c0fe070 2024-03-19T03:14:15.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.498+0000 7fd582238700 1 --2- 172.21.15.59:0/2229576965 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd5680692a0 0x7fd56806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:15.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.498+0000 7fd582238700 1 --2- 172.21.15.59:0/2229576965 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd5680692a0 0x7fd56806b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd578004600 tx=0x7fd57800b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:15.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.503+0000 7fd572ffd700 1 -- 172.21.15.59:0/2229576965 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fd56c058330 con 0x7fd57c0fe070 2024-03-19T03:14:15.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.816+0000 7fd58449c700 1 -- 172.21.15.59:0/2229576965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd57c04bed0 con 0x7fd57c0fe070 2024-03-19T03:14:15.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.817+0000 7fd572ffd700 1 -- 172.21.15.59:0/2229576965 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7fd56c054030 con 0x7fd57c0fe070 2024-03-19T03:14:15.818 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-19T03:14:15.818 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": { 2024-03-19T03:14:15.819 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-700-gb4843bd7 (b4843bd759866f9ee1bdfb1dfa084ec7d085b910) quincy (stable)": 2 2024-03-19T03:14:15.819 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-19T03:14:15.819 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": { 2024-03-19T03:14:15.819 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-700-gb4843bd7 (b4843bd759866f9ee1bdfb1dfa084ec7d085b910) quincy (stable)": 2 2024-03-19T03:14:15.819 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-19T03:14:15.819 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": { 2024-03-19T03:14:15.819 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-700-gb4843bd7 (b4843bd759866f9ee1bdfb1dfa084ec7d085b910) quincy (stable)": 6 2024-03-19T03:14:15.819 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-19T03:14:15.819 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": { 2024-03-19T03:14:15.819 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-700-gb4843bd7 (b4843bd759866f9ee1bdfb1dfa084ec7d085b910) quincy (stable)": 4 2024-03-19T03:14:15.819 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-19T03:14:15.819 INFO:teuthology.orchestra.run.smithi059.stdout: "overall": { 2024-03-19T03:14:15.819 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-700-gb4843bd7 (b4843bd759866f9ee1bdfb1dfa084ec7d085b910) quincy (stable)": 14 2024-03-19T03:14:15.819 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-19T03:14:15.820 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-19T03:14:15.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.819+0000 7fd58449c700 1 -- 172.21.15.59:0/2229576965 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd5680692a0 msgr2=0x7fd56806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:15.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.819+0000 7fd58449c700 1 --2- 172.21.15.59:0/2229576965 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd5680692a0 0x7fd56806b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd578004600 tx=0x7fd57800b040 comp rx=0 tx=0).stop 2024-03-19T03:14:15.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.819+0000 7fd58449c700 1 -- 172.21.15.59:0/2229576965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd57c0fe070 msgr2=0x7fd57c10df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:15.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.819+0000 7fd58449c700 1 --2- 172.21.15.59:0/2229576965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd57c0fe070 0x7fd57c10df10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd56c013870 tx=0x7fd56c009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:15.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.819+0000 7fd58449c700 1 -- 172.21.15.59:0/2229576965 shutdown_connections 2024-03-19T03:14:15.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.819+0000 7fd58449c700 1 --2- 172.21.15.59:0/2229576965 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd5680692a0 0x7fd56806b750 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:15.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.819+0000 7fd58449c700 1 --2- 172.21.15.59:0/2229576965 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd57c0fcf00 0x7fd57c10d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:15.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.819+0000 7fd58449c700 1 --2- 172.21.15.59:0/2229576965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd57c0fe070 0x7fd57c10df10 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:15.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.819+0000 7fd58449c700 1 -- 172.21.15.59:0/2229576965 >> 172.21.15.59:0/2229576965 conn(0x7fd57c0f85a0 msgr2=0x7fd57c0f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:15.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.819+0000 7fd58449c700 1 -- 172.21.15.59:0/2229576965 shutdown_connections 2024-03-19T03:14:15.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.819+0000 7fd58449c700 1 -- 172.21.15.59:0/2229576965 wait complete. 2024-03-19T03:14:15.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.941+0000 7f2897d05700 1 -- 172.21.15.59:0/2293041386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2890066220 msgr2=0x7f28900ffa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:15.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.941+0000 7f2897d05700 1 --2- 172.21.15.59:0/2293041386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2890066220 0x7f28900ffa80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f288c004940 tx=0x7f288c009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:15.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.941+0000 7f2897d05700 1 -- 172.21.15.59:0/2293041386 shutdown_connections 2024-03-19T03:14:15.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.941+0000 7f2897d05700 1 --2- 172.21.15.59:0/2293041386 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28900fffc0 0x7f28901023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:15.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.941+0000 7f2897d05700 1 --2- 172.21.15.59:0/2293041386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2890066220 0x7f28900ffa80 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:15.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.941+0000 7f2897d05700 1 -- 172.21.15.59:0/2293041386 >> 172.21.15.59:0/2293041386 conn(0x7f28900f7430 msgr2=0x7f28900f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:15.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.942+0000 7f2897d05700 1 -- 172.21.15.59:0/2293041386 shutdown_connections 2024-03-19T03:14:15.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.942+0000 7f2897d05700 1 -- 172.21.15.59:0/2293041386 wait complete. 2024-03-19T03:14:15.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.942+0000 7f2897d05700 1 Processor -- start 2024-03-19T03:14:15.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.942+0000 7f2897d05700 1 -- start start 2024-03-19T03:14:15.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.942+0000 7f2897d05700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2890066220 0x7f289010da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:15.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.942+0000 7f2897d05700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28900fffc0 0x7f289010dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:15.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.942+0000 7f2897d05700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289010e5b0 con 0x7f2890066220 2024-03-19T03:14:15.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.942+0000 7f2897d05700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289010edb0 con 0x7f28900fffc0 2024-03-19T03:14:15.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.942+0000 7f2895aa1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2890066220 0x7f289010da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:15.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.943+0000 7f2895aa1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2890066220 0x7f289010da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52490/0 (socket says 172.21.15.59:52490) 2024-03-19T03:14:15.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.943+0000 7f2895aa1700 1 -- 172.21.15.59:0/3787871050 learned_addr learned my addr 172.21.15.59:0/3787871050 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:14:15.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.943+0000 7f28952a0700 1 --2- 172.21.15.59:0/3787871050 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28900fffc0 0x7f289010dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:15.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.943+0000 7f2895aa1700 1 -- 172.21.15.59:0/3787871050 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28900fffc0 msgr2=0x7f289010dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:15.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.943+0000 7f2895aa1700 1 --2- 172.21.15.59:0/3787871050 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28900fffc0 0x7f289010dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:15.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.943+0000 7f2895aa1700 1 -- 172.21.15.59:0/3787871050 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2880005040 con 0x7f2890066220 2024-03-19T03:14:15.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.943+0000 7f2895aa1700 1 --2- 172.21.15.59:0/3787871050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2890066220 0x7f289010da70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f288c000c00 tx=0x7f288c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:15.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.943+0000 7f28867fc700 1 -- 172.21.15.59:0/3787871050 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f288c00e030 con 0x7f2890066220 2024-03-19T03:14:15.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.943+0000 7f2897d05700 1 -- 172.21.15.59:0/3787871050 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f288c005040 con 0x7f2890066220 2024-03-19T03:14:15.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.943+0000 7f28867fc700 1 -- 172.21.15.59:0/3787871050 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f288c013de0 con 0x7f2890066220 2024-03-19T03:14:15.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.943+0000 7f28867fc700 1 -- 172.21.15.59:0/3787871050 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f288c01ce50 con 0x7f2890066220 2024-03-19T03:14:15.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.943+0000 7f2897d05700 1 -- 172.21.15.59:0/3787871050 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f289010f3e0 con 0x7f2890066220 2024-03-19T03:14:15.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.945+0000 7f2897d05700 1 -- 172.21.15.59:0/3787871050 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f289005f5f0 con 0x7f2890066220 2024-03-19T03:14:15.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.945+0000 7f28867fc700 1 -- 172.21.15.59:0/3787871050 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f288c01c400 con 0x7f2890066220 2024-03-19T03:14:15.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.946+0000 7f28867fc700 1 --2- 172.21.15.59:0/3787871050 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f287c069230 0x7f287c06b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:15.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.946+0000 7f28867fc700 1 -- 172.21.15.59:0/3787871050 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f288c088c70 con 0x7f2890066220 2024-03-19T03:14:15.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.946+0000 7f28952a0700 1 --2- 172.21.15.59:0/3787871050 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f287c069230 0x7f287c06b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:15.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.947+0000 7f28952a0700 1 --2- 172.21.15.59:0/3787871050 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f287c069230 0x7f287c06b6e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f28900666e0 tx=0x7f288000b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:15.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:15.953+0000 7f28867fc700 1 -- 172.21.15.59:0/3787871050 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f288c0589b0 con 0x7f2890066220 2024-03-19T03:14:16.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.237+0000 7f2897d05700 1 -- 172.21.15.59:0/3787871050 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f289010f600 con 0x7f2890066220 2024-03-19T03:14:16.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.237+0000 7f28867fc700 1 -- 172.21.15.59:0/3787871050 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1806 (secure 0 0 0) 0x7f288c058540 con 0x7f2890066220 2024-03-19T03:14:16.239 INFO:teuthology.orchestra.run.smithi059.stdout:e14 2024-03-19T03:14:16.239 INFO:teuthology.orchestra.run.smithi059.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-19T03:14:16.240 INFO:teuthology.orchestra.run.smithi059.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-19T03:14:16.240 INFO:teuthology.orchestra.run.smithi059.stdout:legacy client fscid: 1 2024-03-19T03:14:16.240 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:14:16.240 INFO:teuthology.orchestra.run.smithi059.stdout:Filesystem 'cephfs' (1) 2024-03-19T03:14:16.240 INFO:teuthology.orchestra.run.smithi059.stdout:fs_name cephfs 2024-03-19T03:14:16.240 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 13 2024-03-19T03:14:16.240 INFO:teuthology.orchestra.run.smithi059.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-19T03:14:16.240 INFO:teuthology.orchestra.run.smithi059.stdout:created 2024-03-19T03:13:17.039133+0000 2024-03-19T03:14:16.240 INFO:teuthology.orchestra.run.smithi059.stdout:modified 2024-03-19T03:13:32.433876+0000 2024-03-19T03:14:16.240 INFO:teuthology.orchestra.run.smithi059.stdout:tableserver 0 2024-03-19T03:14:16.240 INFO:teuthology.orchestra.run.smithi059.stdout:root 0 2024-03-19T03:14:16.241 INFO:teuthology.orchestra.run.smithi059.stdout:session_timeout 60 2024-03-19T03:14:16.241 INFO:teuthology.orchestra.run.smithi059.stdout:session_autoclose 300 2024-03-19T03:14:16.241 INFO:teuthology.orchestra.run.smithi059.stdout:max_file_size 1099511627776 2024-03-19T03:14:16.241 INFO:teuthology.orchestra.run.smithi059.stdout:required_client_features {} 2024-03-19T03:14:16.241 INFO:teuthology.orchestra.run.smithi059.stdout:last_failure 0 2024-03-19T03:14:16.241 INFO:teuthology.orchestra.run.smithi059.stdout:last_failure_osd_epoch 0 2024-03-19T03:14:16.241 INFO:teuthology.orchestra.run.smithi059.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-19T03:14:16.241 INFO:teuthology.orchestra.run.smithi059.stdout:max_mds 1 2024-03-19T03:14:16.241 INFO:teuthology.orchestra.run.smithi059.stdout:in 0 2024-03-19T03:14:16.241 INFO:teuthology.orchestra.run.smithi059.stdout:up {0=14468} 2024-03-19T03:14:16.241 INFO:teuthology.orchestra.run.smithi059.stdout:failed 2024-03-19T03:14:16.241 INFO:teuthology.orchestra.run.smithi059.stdout:damaged 2024-03-19T03:14:16.242 INFO:teuthology.orchestra.run.smithi059.stdout:stopped 2024-03-19T03:14:16.242 INFO:teuthology.orchestra.run.smithi059.stdout:data_pools [3] 2024-03-19T03:14:16.242 INFO:teuthology.orchestra.run.smithi059.stdout:metadata_pool 2 2024-03-19T03:14:16.242 INFO:teuthology.orchestra.run.smithi059.stdout:inline_data enabled 2024-03-19T03:14:16.242 INFO:teuthology.orchestra.run.smithi059.stdout:balancer 2024-03-19T03:14:16.242 INFO:teuthology.orchestra.run.smithi059.stdout:standby_count_wanted 1 2024-03-19T03:14:16.242 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi059.hmylwa{0:14468} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.59:6826/2936511711,v1:172.21.15.59:6827/2936511711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-19T03:14:16.242 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:14:16.242 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:14:16.242 INFO:teuthology.orchestra.run.smithi059.stdout:Standby daemons: 2024-03-19T03:14:16.242 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:14:16.243 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi059.doeuxg{-1:14482} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.59:6828/503726080,v1:172.21.15.59:6829/503726080] compat {c=[1],r=[1],i=[7ff]}] 2024-03-19T03:14:16.243 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi176.bochhq{-1:24279} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.176:6824/193753714,v1:172.21.15.176:6825/193753714] compat {c=[1],r=[1],i=[7ff]}] 2024-03-19T03:14:16.243 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi176.gisnqc{-1:24289} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.176:6826/4058958323,v1:172.21.15.176:6827/4058958323] compat {c=[1],r=[1],i=[7ff]}] 2024-03-19T03:14:16.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.239+0000 7f2897d05700 1 -- 172.21.15.59:0/3787871050 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f287c069230 msgr2=0x7f287c06b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:16.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.239+0000 7f2897d05700 1 --2- 172.21.15.59:0/3787871050 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f287c069230 0x7f287c06b6e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f28900666e0 tx=0x7f288000b040 comp rx=0 tx=0).stop 2024-03-19T03:14:16.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.239+0000 7f2897d05700 1 -- 172.21.15.59:0/3787871050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2890066220 msgr2=0x7f289010da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:16.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.239+0000 7f2897d05700 1 --2- 172.21.15.59:0/3787871050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2890066220 0x7f289010da70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f288c000c00 tx=0x7f288c009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:16.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.240+0000 7f2897d05700 1 -- 172.21.15.59:0/3787871050 shutdown_connections 2024-03-19T03:14:16.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.240+0000 7f2897d05700 1 --2- 172.21.15.59:0/3787871050 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f287c069230 0x7f287c06b6e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:16.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.240+0000 7f2897d05700 1 --2- 172.21.15.59:0/3787871050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2890066220 0x7f289010da70 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:16.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.240+0000 7f2897d05700 1 --2- 172.21.15.59:0/3787871050 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28900fffc0 0x7f289010dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:16.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.240+0000 7f2897d05700 1 -- 172.21.15.59:0/3787871050 >> 172.21.15.59:0/3787871050 conn(0x7f28900f7430 msgr2=0x7f28900f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:16.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.240+0000 7f2897d05700 1 -- 172.21.15.59:0/3787871050 shutdown_connections 2024-03-19T03:14:16.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.240+0000 7f2897d05700 1 -- 172.21.15.59:0/3787871050 wait complete. 2024-03-19T03:14:16.245 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 14 2024-03-19T03:14:16.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.362+0000 7fcddb1fd700 1 -- 172.21.15.59:0/1860895643 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdd40662e0 msgr2=0x7fcdd40ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:16.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.362+0000 7fcddb1fd700 1 --2- 172.21.15.59:0/1860895643 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdd40662e0 0x7fcdd40ff9c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8004940 tx=0x7fcdc8009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:16.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.362+0000 7fcddb1fd700 1 -- 172.21.15.59:0/1860895643 shutdown_connections 2024-03-19T03:14:16.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.362+0000 7fcddb1fd700 1 --2- 172.21.15.59:0/1860895643 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcdd40fff00 0x7fcdd4102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:16.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.362+0000 7fcddb1fd700 1 --2- 172.21.15.59:0/1860895643 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdd40662e0 0x7fcdd40ff9c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:16.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.362+0000 7fcddb1fd700 1 -- 172.21.15.59:0/1860895643 >> 172.21.15.59:0/1860895643 conn(0x7fcdd40f7430 msgr2=0x7fcdd40f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:16.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.362+0000 7fcddb1fd700 1 -- 172.21.15.59:0/1860895643 shutdown_connections 2024-03-19T03:14:16.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.362+0000 7fcddb1fd700 1 -- 172.21.15.59:0/1860895643 wait complete. 2024-03-19T03:14:16.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.363+0000 7fcddb1fd700 1 Processor -- start 2024-03-19T03:14:16.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.363+0000 7fcddb1fd700 1 -- start start 2024-03-19T03:14:16.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.363+0000 7fcddb1fd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdd40662e0 0x7fcdd410da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:16.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.363+0000 7fcddb1fd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcdd40fff00 0x7fcdd410df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:16.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.363+0000 7fcddb1fd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd410e540 con 0x7fcdd40662e0 2024-03-19T03:14:16.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.363+0000 7fcddb1fd700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd410ed40 con 0x7fcdd40fff00 2024-03-19T03:14:16.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.363+0000 7fcdd8f99700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdd40662e0 0x7fcdd410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:16.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.363+0000 7fcdd3fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcdd40fff00 0x7fcdd410df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:16.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.363+0000 7fcdd8f99700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdd40662e0 0x7fcdd410da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52502/0 (socket says 172.21.15.59:52502) 2024-03-19T03:14:16.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.363+0000 7fcdd8f99700 1 -- 172.21.15.59:0/3507984468 learned_addr learned my addr 172.21.15.59:0/3507984468 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:14:16.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.364+0000 7fcdd8f99700 1 -- 172.21.15.59:0/3507984468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcdd40fff00 msgr2=0x7fcdd410df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:16.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.364+0000 7fcdd8f99700 1 --2- 172.21.15.59:0/3507984468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcdd40fff00 0x7fcdd410df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:16.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.364+0000 7fcdd8f99700 1 -- 172.21.15.59:0/3507984468 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc8005040 con 0x7fcdd40662e0 2024-03-19T03:14:16.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.364+0000 7fcdd8f99700 1 --2- 172.21.15.59:0/3507984468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdd40662e0 0x7fcdd410da00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8004090 tx=0x7fcdc8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:16.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.364+0000 7fcdd17fa700 1 -- 172.21.15.59:0/3507984468 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdc8021070 con 0x7fcdd40662e0 2024-03-19T03:14:16.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.364+0000 7fcddb1fd700 1 -- 172.21.15.59:0/3507984468 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd410efc0 con 0x7fcdd40662e0 2024-03-19T03:14:16.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.364+0000 7fcdd17fa700 1 -- 172.21.15.59:0/3507984468 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcdc801c400 con 0x7fcdd40662e0 2024-03-19T03:14:16.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.364+0000 7fcdd17fa700 1 -- 172.21.15.59:0/3507984468 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdc800e070 con 0x7fcdd40662e0 2024-03-19T03:14:16.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.364+0000 7fcddb1fd700 1 -- 172.21.15.59:0/3507984468 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd410f4a0 con 0x7fcdd40662e0 2024-03-19T03:14:16.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.365+0000 7fcddb1fd700 1 -- 172.21.15.59:0/3507984468 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd405f5f0 con 0x7fcdd40662e0 2024-03-19T03:14:16.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.366+0000 7fcdd17fa700 1 -- 172.21.15.59:0/3507984468 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fcdc801ca50 con 0x7fcdd40662e0 2024-03-19T03:14:16.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.366+0000 7fcdd17fa700 1 --2- 172.21.15.59:0/3507984468 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fcdc40692b0 0x7fcdc406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:16.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.367+0000 7fcdd17fa700 1 -- 172.21.15.59:0/3507984468 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fcdc8088270 con 0x7fcdd40662e0 2024-03-19T03:14:16.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.367+0000 7fcdd3fff700 1 --2- 172.21.15.59:0/3507984468 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fcdc40692b0 0x7fcdc406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:16.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.367+0000 7fcdd3fff700 1 --2- 172.21.15.59:0/3507984468 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fcdc40692b0 0x7fcdc406b760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0009b80 tx=0x7fcdc000b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:16.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.371+0000 7fcdd17fa700 1 -- 172.21.15.59:0/3507984468 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fcdc80580d0 con 0x7fcdd40662e0 2024-03-19T03:14:16.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:16 smithi176 ceph-mon[36450]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:14:16.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:16 smithi176 ceph-mon[36450]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:14:16.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:16 smithi176 ceph-mon[36450]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-19T03:14:16.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:16 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/2229576965' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-19T03:14:16.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:16 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:14:16.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:16 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/3787871050' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-19T03:14:16.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:16 smithi059 ceph-mon[29133]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:14:16.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:16 smithi059 ceph-mon[29133]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:14:16.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:16 smithi059 ceph-mon[29133]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-19T03:14:16.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:16 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/2229576965' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-19T03:14:16.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:16 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:14:16.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:16 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/3787871050' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-19T03:14:16.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.648+0000 7fcddb1fd700 1 -- 172.21.15.59:0/3507984468 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdd410f7d0 con 0x7fcdc40692b0 2024-03-19T03:14:16.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.650+0000 7fcdd17fa700 1 -- 172.21.15.59:0/3507984468 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7fcdd410f7d0 con 0x7fcdc40692b0 2024-03-19T03:14:16.651 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-19T03:14:16.651 INFO:teuthology.orchestra.run.smithi059.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810", 2024-03-19T03:14:16.651 INFO:teuthology.orchestra.run.smithi059.stdout: "in_progress": true, 2024-03-19T03:14:16.651 INFO:teuthology.orchestra.run.smithi059.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-19T03:14:16.652 INFO:teuthology.orchestra.run.smithi059.stdout: "services_complete": [], 2024-03-19T03:14:16.652 INFO:teuthology.orchestra.run.smithi059.stdout: "progress": "", 2024-03-19T03:14:16.652 INFO:teuthology.orchestra.run.smithi059.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 image", 2024-03-19T03:14:16.652 INFO:teuthology.orchestra.run.smithi059.stdout: "is_paused": false 2024-03-19T03:14:16.652 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-19T03:14:16.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.652+0000 7fcddb1fd700 1 -- 172.21.15.59:0/3507984468 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fcdc40692b0 msgr2=0x7fcdc406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:16.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.652+0000 7fcddb1fd700 1 --2- 172.21.15.59:0/3507984468 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fcdc40692b0 0x7fcdc406b760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0009b80 tx=0x7fcdc000b040 comp rx=0 tx=0).stop 2024-03-19T03:14:16.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.652+0000 7fcddb1fd700 1 -- 172.21.15.59:0/3507984468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdd40662e0 msgr2=0x7fcdd410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:16.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.652+0000 7fcddb1fd700 1 --2- 172.21.15.59:0/3507984468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdd40662e0 0x7fcdd410da00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8004090 tx=0x7fcdc8009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:16.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.652+0000 7fcddb1fd700 1 -- 172.21.15.59:0/3507984468 shutdown_connections 2024-03-19T03:14:16.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.652+0000 7fcddb1fd700 1 --2- 172.21.15.59:0/3507984468 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fcdc40692b0 0x7fcdc406b760 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:16.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.652+0000 7fcddb1fd700 1 --2- 172.21.15.59:0/3507984468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdd40662e0 0x7fcdd410da00 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:16.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.652+0000 7fcddb1fd700 1 --2- 172.21.15.59:0/3507984468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcdd40fff00 0x7fcdd410df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:16.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.652+0000 7fcddb1fd700 1 -- 172.21.15.59:0/3507984468 >> 172.21.15.59:0/3507984468 conn(0x7fcdd40f7430 msgr2=0x7fcdd40f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:16.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.652+0000 7fcddb1fd700 1 -- 172.21.15.59:0/3507984468 shutdown_connections 2024-03-19T03:14:16.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.652+0000 7fcddb1fd700 1 -- 172.21.15.59:0/3507984468 wait complete. 2024-03-19T03:14:16.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.772+0000 7fb858df7700 1 -- 172.21.15.59:0/1674873233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8540fc2b0 msgr2=0x7fb8540fc6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:16.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.772+0000 7fb858df7700 1 --2- 172.21.15.59:0/1674873233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8540fc2b0 0x7fb8540fc6c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb844004940 tx=0x7fb844009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:16.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.772+0000 7fb858df7700 1 -- 172.21.15.59:0/1674873233 shutdown_connections 2024-03-19T03:14:16.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.772+0000 7fb858df7700 1 --2- 172.21.15.59:0/1674873233 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8540fcc00 0x7fb8540fd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:16.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.772+0000 7fb858df7700 1 --2- 172.21.15.59:0/1674873233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8540fc2b0 0x7fb8540fc6c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:16.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.772+0000 7fb858df7700 1 -- 172.21.15.59:0/1674873233 >> 172.21.15.59:0/1674873233 conn(0x7fb8540f6ba0 msgr2=0x7fb8540f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:16.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.772+0000 7fb858df7700 1 -- 172.21.15.59:0/1674873233 shutdown_connections 2024-03-19T03:14:16.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.772+0000 7fb858df7700 1 -- 172.21.15.59:0/1674873233 wait complete. 2024-03-19T03:14:16.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.772+0000 7fb858df7700 1 Processor -- start 2024-03-19T03:14:16.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.773+0000 7fb858df7700 1 -- start start 2024-03-19T03:14:16.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.773+0000 7fb858df7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8540fc2b0 0x7fb8540ffe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:16.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.773+0000 7fb858df7700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8540fcc00 0x7fb8540fe3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:16.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.773+0000 7fb858df7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8540fe940 con 0x7fb8540fc2b0 2024-03-19T03:14:16.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.773+0000 7fb858df7700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8540fea80 con 0x7fb8540fcc00 2024-03-19T03:14:16.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.773+0000 7fb85259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8540fc2b0 0x7fb8540ffe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:16.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.773+0000 7fb85259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8540fc2b0 0x7fb8540ffe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52520/0 (socket says 172.21.15.59:52520) 2024-03-19T03:14:16.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.773+0000 7fb85259c700 1 -- 172.21.15.59:0/3270020870 learned_addr learned my addr 172.21.15.59:0/3270020870 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:14:16.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.773+0000 7fb851d9b700 1 --2- 172.21.15.59:0/3270020870 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8540fcc00 0x7fb8540fe3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:16.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.773+0000 7fb85259c700 1 -- 172.21.15.59:0/3270020870 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8540fcc00 msgr2=0x7fb8540fe3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:16.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.773+0000 7fb85259c700 1 --2- 172.21.15.59:0/3270020870 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8540fcc00 0x7fb8540fe3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:16.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.773+0000 7fb85259c700 1 -- 172.21.15.59:0/3270020870 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb844005040 con 0x7fb8540fc2b0 2024-03-19T03:14:16.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.773+0000 7fb85259c700 1 --2- 172.21.15.59:0/3270020870 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8540fc2b0 0x7fb8540ffe90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb84400b0d0 tx=0x7fb844009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:16.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.774+0000 7fb842ffd700 1 -- 172.21.15.59:0/3270020870 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb84400e070 con 0x7fb8540fc2b0 2024-03-19T03:14:16.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.774+0000 7fb842ffd700 1 -- 172.21.15.59:0/3270020870 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb844013de0 con 0x7fb8540fc2b0 2024-03-19T03:14:16.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.774+0000 7fb858df7700 1 -- 172.21.15.59:0/3270020870 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8540fed00 con 0x7fb8540fc2b0 2024-03-19T03:14:16.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.774+0000 7fb842ffd700 1 -- 172.21.15.59:0/3270020870 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb84400be30 con 0x7fb8540fc2b0 2024-03-19T03:14:16.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.774+0000 7fb858df7700 1 -- 172.21.15.59:0/3270020870 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb854115ab0 con 0x7fb8540fc2b0 2024-03-19T03:14:16.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.775+0000 7fb858df7700 1 -- 172.21.15.59:0/3270020870 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb854101c00 con 0x7fb8540fc2b0 2024-03-19T03:14:16.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.775+0000 7fb842ffd700 1 -- 172.21.15.59:0/3270020870 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb84400b680 con 0x7fb8540fc2b0 2024-03-19T03:14:16.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.776+0000 7fb842ffd700 1 --2- 172.21.15.59:0/3270020870 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb83c0692b0 0x7fb83c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:16.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.776+0000 7fb842ffd700 1 -- 172.21.15.59:0/3270020870 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fb844087e50 con 0x7fb8540fc2b0 2024-03-19T03:14:16.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.776+0000 7fb851d9b700 1 --2- 172.21.15.59:0/3270020870 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb83c0692b0 0x7fb83c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:16.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.776+0000 7fb851d9b700 1 --2- 172.21.15.59:0/3270020870 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb83c0692b0 0x7fb83c06b760 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb854100470 tx=0x7fb84800b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:16.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:16.782+0000 7fb842ffd700 1 -- 172.21.15.59:0/3270020870 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fb844054220 con 0x7fb8540fc2b0 2024-03-19T03:14:17.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:17.157+0000 7fb858df7700 1 -- 172.21.15.59:0/3270020870 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb8541048a0 con 0x7fb8540fc2b0 2024-03-19T03:14:17.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:17.157+0000 7fb842ffd700 1 -- 172.21.15.59:0/3270020870 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fb844057840 con 0x7fb8540fc2b0 2024-03-19T03:14:17.159 INFO:teuthology.orchestra.run.smithi059.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-03-19T03:14:17.159 INFO:teuthology.orchestra.run.smithi059.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-03-19T03:14:17.159 INFO:teuthology.orchestra.run.smithi059.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-03-19T03:14:17.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:17.159+0000 7fb858df7700 1 -- 172.21.15.59:0/3270020870 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb83c0692b0 msgr2=0x7fb83c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:17.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:17.159+0000 7fb858df7700 1 --2- 172.21.15.59:0/3270020870 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb83c0692b0 0x7fb83c06b760 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb854100470 tx=0x7fb84800b040 comp rx=0 tx=0).stop 2024-03-19T03:14:17.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:17.159+0000 7fb858df7700 1 -- 172.21.15.59:0/3270020870 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8540fc2b0 msgr2=0x7fb8540ffe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:17.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:17.159+0000 7fb858df7700 1 --2- 172.21.15.59:0/3270020870 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8540fc2b0 0x7fb8540ffe90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb84400b0d0 tx=0x7fb844009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:17.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:17.159+0000 7fb858df7700 1 -- 172.21.15.59:0/3270020870 shutdown_connections 2024-03-19T03:14:17.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:17.159+0000 7fb858df7700 1 --2- 172.21.15.59:0/3270020870 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fb83c0692b0 0x7fb83c06b760 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:17.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:17.159+0000 7fb858df7700 1 --2- 172.21.15.59:0/3270020870 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8540fc2b0 0x7fb8540ffe90 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:17.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:17.159+0000 7fb858df7700 1 --2- 172.21.15.59:0/3270020870 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8540fcc00 0x7fb8540fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:17.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:17.159+0000 7fb858df7700 1 -- 172.21.15.59:0/3270020870 >> 172.21.15.59:0/3270020870 conn(0x7fb8540f6ba0 msgr2=0x7fb8540fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:17.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:17.160+0000 7fb858df7700 1 -- 172.21.15.59:0/3270020870 shutdown_connections 2024-03-19T03:14:17.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:17.160+0000 7fb858df7700 1 -- 172.21.15.59:0/3270020870 wait complete. 2024-03-19T03:14:17.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:17 smithi059 ceph-mon[29133]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:14:17.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:17 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/3270020870' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-19T03:14:17.627 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:17 smithi176 ceph-mon[36450]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:14:17.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:17 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/3270020870' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-19T03:14:18.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:18 smithi059 ceph-mon[29133]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:14:18.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:18 smithi059 ceph-mon[29133]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-19T03:14:18.627 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:18 smithi176 ceph-mon[36450]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:14:18.628 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:18 smithi176 ceph-mon[36450]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-19T03:14:19.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:19 smithi059 ceph-mon[29133]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:14:19.627 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:19 smithi176 ceph-mon[36450]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:14:21.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:21 smithi059 ceph-mon[29133]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:14:21.877 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:21 smithi176 ceph-mon[36450]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:14:24.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:24 smithi059 ceph-mon[29133]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:14:24.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:24 smithi176 ceph-mon[36450]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:14:26.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:26 smithi059 ceph-mon[29133]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:14:26.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:26 smithi176 ceph-mon[36450]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:14:28.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:28 smithi059 ceph-mon[29133]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:14:28.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:28 smithi176 ceph-mon[36450]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:14:30.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:30 smithi059 ceph-mon[29133]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:14:30.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:30 smithi176 ceph-mon[36450]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:14:31.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:31 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:14:31.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:31 smithi059 ceph-mon[29133]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:14:31.877 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:31 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:14:31.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:31 smithi176 ceph-mon[36450]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-19T03:14:34.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:34 smithi059 ceph-mon[29133]: pgmap v214: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-03-19T03:14:34.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:34 smithi176 ceph-mon[36450]: pgmap v214: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-03-19T03:14:36.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:36 smithi059 ceph-mon[29133]: pgmap v215: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-03-19T03:14:36.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:36 smithi176 ceph-mon[36450]: pgmap v215: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-03-19T03:14:38.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:38 smithi059 ceph-mon[29133]: pgmap v216: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-03-19T03:14:38.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:38 smithi176 ceph-mon[36450]: pgmap v216: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-03-19T03:14:40.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:40 smithi059 ceph-mon[29133]: pgmap v217: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-03-19T03:14:40.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:40 smithi176 ceph-mon[36450]: pgmap v217: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-03-19T03:14:41.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:41 smithi059 ceph-mon[29133]: pgmap v218: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-19T03:14:41.877 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:41 smithi176 ceph-mon[36450]: pgmap v218: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-19T03:14:43.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:43 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:14:43.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:43 smithi059 ceph-mon[29133]: Upgrade: Target is version 19.0.0-2206-gb6ae2491 (unknown) 2024-03-19T03:14:43.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:43 smithi059 ceph-mon[29133]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, digests ['quay.ceph.io/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639'] 2024-03-19T03:14:43.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:43 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:14:43.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:43 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:14:43.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:43 smithi059 ceph-mon[29133]: Upgrade: Need to upgrade myself (mgr.smithi059.azpecc) 2024-03-19T03:14:43.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:43 smithi059 ceph-mon[29133]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 on smithi176 2024-03-19T03:14:43.840 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:43 smithi059 ceph-mon[29133]: pgmap v219: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-19T03:14:43.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:43 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:14:43.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:43 smithi176 ceph-mon[36450]: Upgrade: Target is version 19.0.0-2206-gb6ae2491 (unknown) 2024-03-19T03:14:43.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:43 smithi176 ceph-mon[36450]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, digests ['quay.ceph.io/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639'] 2024-03-19T03:14:43.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:43 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:14:43.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:43 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:14:43.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:43 smithi176 ceph-mon[36450]: Upgrade: Need to upgrade myself (mgr.smithi059.azpecc) 2024-03-19T03:14:43.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:43 smithi176 ceph-mon[36450]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 on smithi176 2024-03-19T03:14:43.878 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:43 smithi176 ceph-mon[36450]: pgmap v219: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-19T03:14:46.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:46 smithi059 ceph-mon[29133]: pgmap v220: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-19T03:14:46.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:46 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:14:46.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:46 smithi176 ceph-mon[36450]: pgmap v220: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-19T03:14:46.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:46 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:14:46.658 INFO:tasks.workunit.client.1.smithi176.stderr:Note: switching to 'e48b1f6c0608b2615e1890cc46286c3877d6e910'. 2024-03-19T03:14:46.658 INFO:tasks.workunit.client.1.smithi176.stderr: 2024-03-19T03:14:46.658 INFO:tasks.workunit.client.1.smithi176.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-19T03:14:46.658 INFO:tasks.workunit.client.1.smithi176.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-19T03:14:46.658 INFO:tasks.workunit.client.1.smithi176.stderr:state without impacting any branches by switching back to a branch. 2024-03-19T03:14:46.659 INFO:tasks.workunit.client.1.smithi176.stderr: 2024-03-19T03:14:46.659 INFO:tasks.workunit.client.1.smithi176.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-19T03:14:46.659 INFO:tasks.workunit.client.1.smithi176.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-19T03:14:46.659 INFO:tasks.workunit.client.1.smithi176.stderr: 2024-03-19T03:14:46.659 INFO:tasks.workunit.client.1.smithi176.stderr: git switch -c 2024-03-19T03:14:46.659 INFO:tasks.workunit.client.1.smithi176.stderr: 2024-03-19T03:14:46.659 INFO:tasks.workunit.client.1.smithi176.stderr:Or undo this operation with: 2024-03-19T03:14:46.659 INFO:tasks.workunit.client.1.smithi176.stderr: 2024-03-19T03:14:46.659 INFO:tasks.workunit.client.1.smithi176.stderr: git switch - 2024-03-19T03:14:46.659 INFO:tasks.workunit.client.1.smithi176.stderr: 2024-03-19T03:14:46.659 INFO:tasks.workunit.client.1.smithi176.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-19T03:14:46.659 INFO:tasks.workunit.client.1.smithi176.stderr: 2024-03-19T03:14:46.659 INFO:tasks.workunit.client.1.smithi176.stderr:HEAD is now at e48b1f6c060 testfix 2024-03-19T03:14:46.665 DEBUG:teuthology.orchestra.run.smithi176:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-03-19T03:14:46.733 INFO:tasks.workunit.client.1.smithi176.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-19T03:14:46.734 INFO:tasks.workunit.client.1.smithi176.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-19T03:14:46.734 INFO:tasks.workunit.client.1.smithi176.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-19T03:14:47.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.267+0000 7f2202cae700 1 -- 172.21.15.59:0/2537672515 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21fc066af0 msgr2=0x7f21fc0ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:47.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.267+0000 7f2202cae700 1 --2- 172.21.15.59:0/2537672515 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21fc066af0 0x7f21fc0ffa50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f21f0004940 tx=0x7f21f0009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:47.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.267+0000 7f2202cae700 1 -- 172.21.15.59:0/2537672515 shutdown_connections 2024-03-19T03:14:47.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.267+0000 7f2202cae700 1 --2- 172.21.15.59:0/2537672515 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21fc0fff90 0x7f21fc102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:47.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.267+0000 7f2202cae700 1 --2- 172.21.15.59:0/2537672515 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21fc066af0 0x7f21fc0ffa50 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:47.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.267+0000 7f2202cae700 1 -- 172.21.15.59:0/2537672515 >> 172.21.15.59:0/2537672515 conn(0x7f21fc0f7480 msgr2=0x7f21fc0f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:47.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.267+0000 7f2202cae700 1 -- 172.21.15.59:0/2537672515 shutdown_connections 2024-03-19T03:14:47.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.267+0000 7f2202cae700 1 -- 172.21.15.59:0/2537672515 wait complete. 2024-03-19T03:14:47.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.268+0000 7f2202cae700 1 Processor -- start 2024-03-19T03:14:47.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.268+0000 7f2202cae700 1 -- start start 2024-03-19T03:14:47.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.268+0000 7f2202cae700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21fc066af0 0x7f21fc0fd430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:47.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.268+0000 7f2202cae700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21fc0fff90 0x7f21fc0fb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:47.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.268+0000 7f2202cae700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc0fbee0 con 0x7f21fc0fff90 2024-03-19T03:14:47.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.268+0000 7f2202cae700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc0fc020 con 0x7f21fc066af0 2024-03-19T03:14:47.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.268+0000 7f21fbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21fc0fff90 0x7f21fc0fb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:47.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.268+0000 7f21fbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21fc0fff90 0x7f21fc0fb950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60416/0 (socket says 172.21.15.59:60416) 2024-03-19T03:14:47.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.268+0000 7f21fbfff700 1 -- 172.21.15.59:0/4037558545 learned_addr learned my addr 172.21.15.59:0/4037558545 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:14:47.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.268+0000 7f2200a4a700 1 --2- 172.21.15.59:0/4037558545 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21fc066af0 0x7f21fc0fd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:47.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.269+0000 7f21fbfff700 1 -- 172.21.15.59:0/4037558545 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21fc066af0 msgr2=0x7f21fc0fd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:47.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.269+0000 7f21fbfff700 1 --2- 172.21.15.59:0/4037558545 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21fc066af0 0x7f21fc0fd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:47.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.269+0000 7f21fbfff700 1 -- 172.21.15.59:0/4037558545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f0005040 con 0x7f21fc0fff90 2024-03-19T03:14:47.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.269+0000 7f21fbfff700 1 --2- 172.21.15.59:0/4037558545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21fc0fff90 0x7f21fc0fb950 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f21e800eb40 tx=0x7f21e800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:47.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.269+0000 7f21f97fa700 1 -- 172.21.15.59:0/4037558545 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21e8007cc0 con 0x7f21fc0fff90 2024-03-19T03:14:47.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.269+0000 7f2202cae700 1 -- 172.21.15.59:0/4037558545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21fc0fc2a0 con 0x7f21fc0fff90 2024-03-19T03:14:47.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.269+0000 7f21f97fa700 1 -- 172.21.15.59:0/4037558545 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f21e800f040 con 0x7f21fc0fff90 2024-03-19T03:14:47.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.269+0000 7f21f97fa700 1 -- 172.21.15.59:0/4037558545 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21e800ca10 con 0x7f21fc0fff90 2024-03-19T03:14:47.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.269+0000 7f2202cae700 1 -- 172.21.15.59:0/4037558545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21fc115a50 con 0x7f21fc0fff90 2024-03-19T03:14:47.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.271+0000 7f2202cae700 1 -- 172.21.15.59:0/4037558545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21fc05f5f0 con 0x7f21fc0fff90 2024-03-19T03:14:47.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.271+0000 7f21f97fa700 1 -- 172.21.15.59:0/4037558545 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f21e800cc80 con 0x7f21fc0fff90 2024-03-19T03:14:47.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.272+0000 7f21f97fa700 1 --2- 172.21.15.59:0/4037558545 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f21ec0692b0 0x7f21ec06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:47.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.272+0000 7f21f97fa700 1 -- 172.21.15.59:0/4037558545 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f21e8086d80 con 0x7f21fc0fff90 2024-03-19T03:14:47.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.272+0000 7f2200a4a700 1 --2- 172.21.15.59:0/4037558545 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f21ec0692b0 0x7f21ec06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:47.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.272+0000 7f2200a4a700 1 --2- 172.21.15.59:0/4037558545 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f21ec0692b0 0x7f21ec06b760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f21f0013740 tx=0x7f21f0009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:47.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.279+0000 7f21f97fa700 1 -- 172.21.15.59:0/4037558545 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f21e804de50 con 0x7f21fc0fff90 2024-03-19T03:14:47.478 INFO:tasks.workunit.client.1.smithi176.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-19T03:14:47.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.542+0000 7f2202cae700 1 -- 172.21.15.59:0/4037558545 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f21fc0f8f70 con 0x7f21ec0692b0 2024-03-19T03:14:47.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.543+0000 7f21f97fa700 1 -- 172.21.15.59:0/4037558545 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f21fc0f8f70 con 0x7f21ec0692b0 2024-03-19T03:14:47.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.545+0000 7f2202cae700 1 -- 172.21.15.59:0/4037558545 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f21ec0692b0 msgr2=0x7f21ec06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:47.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.545+0000 7f2202cae700 1 --2- 172.21.15.59:0/4037558545 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f21ec0692b0 0x7f21ec06b760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f21f0013740 tx=0x7f21f0009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:47.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.545+0000 7f2202cae700 1 -- 172.21.15.59:0/4037558545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21fc0fff90 msgr2=0x7f21fc0fb950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:47.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.545+0000 7f2202cae700 1 --2- 172.21.15.59:0/4037558545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21fc0fff90 0x7f21fc0fb950 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f21e800eb40 tx=0x7f21e800b040 comp rx=0 tx=0).stop 2024-03-19T03:14:47.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.546+0000 7f2202cae700 1 -- 172.21.15.59:0/4037558545 shutdown_connections 2024-03-19T03:14:47.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.546+0000 7f2202cae700 1 --2- 172.21.15.59:0/4037558545 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f21ec0692b0 0x7f21ec06b760 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:47.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.546+0000 7f2202cae700 1 --2- 172.21.15.59:0/4037558545 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21fc066af0 0x7f21fc0fd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:47.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.546+0000 7f2202cae700 1 --2- 172.21.15.59:0/4037558545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21fc0fff90 0x7f21fc0fb950 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:47.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.546+0000 7f2202cae700 1 -- 172.21.15.59:0/4037558545 >> 172.21.15.59:0/4037558545 conn(0x7f21fc0f7480 msgr2=0x7f21fc0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:47.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.546+0000 7f2202cae700 1 -- 172.21.15.59:0/4037558545 shutdown_connections 2024-03-19T03:14:47.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.546+0000 7f2202cae700 1 -- 172.21.15.59:0/4037558545 wait complete. 2024-03-19T03:14:47.556 INFO:tasks.workunit.client.1.smithi176.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-19T03:14:47.559 INFO:teuthology.orchestra.run.smithi059.stdout:true 2024-03-19T03:14:47.602 INFO:tasks.workunit.client.1.smithi176.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-19T03:14:47.603 INFO:tasks.workunit.client.1.smithi176.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-19T03:14:47.603 INFO:tasks.workunit.client.1.smithi176.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-19T03:14:47.643 INFO:tasks.workunit.client.1.smithi176.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-19T03:14:47.646 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-19T03:14:47.646 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-03-19T03:14:47.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.653+0000 7fed71ceb700 1 -- 172.21.15.59:0/869858717 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed6c0fff00 msgr2=0x7fed6c102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:47.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.653+0000 7fed71ceb700 1 --2- 172.21.15.59:0/869858717 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed6c0fff00 0x7fed6c102320 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fed60004940 tx=0x7fed60009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:47.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.653+0000 7fed71ceb700 1 -- 172.21.15.59:0/869858717 shutdown_connections 2024-03-19T03:14:47.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.653+0000 7fed71ceb700 1 --2- 172.21.15.59:0/869858717 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed6c0fff00 0x7fed6c102320 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:47.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.653+0000 7fed71ceb700 1 --2- 172.21.15.59:0/869858717 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed6c0662e0 0x7fed6c0ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:47.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.653+0000 7fed71ceb700 1 -- 172.21.15.59:0/869858717 >> 172.21.15.59:0/869858717 conn(0x7fed6c0f7430 msgr2=0x7fed6c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:47.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.654+0000 7fed71ceb700 1 -- 172.21.15.59:0/869858717 shutdown_connections 2024-03-19T03:14:47.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.654+0000 7fed71ceb700 1 -- 172.21.15.59:0/869858717 wait complete. 2024-03-19T03:14:47.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.654+0000 7fed71ceb700 1 Processor -- start 2024-03-19T03:14:47.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.654+0000 7fed71ceb700 1 -- start start 2024-03-19T03:14:47.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.654+0000 7fed71ceb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed6c0662e0 0x7fed6c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:47.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.655+0000 7fed71ceb700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed6c0fff00 0x7fed6c0fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:47.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.655+0000 7fed71ceb700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c0fbe50 con 0x7fed6c0662e0 2024-03-19T03:14:47.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.655+0000 7fed71ceb700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c0fbf90 con 0x7fed6c0fff00 2024-03-19T03:14:47.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.655+0000 7fed6affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed6c0fff00 0x7fed6c0fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:47.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.655+0000 7fed6affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed6c0fff00 0x7fed6c0fb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.59:36138/0 (socket says 172.21.15.59:36138) 2024-03-19T03:14:47.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.655+0000 7fed6affd700 1 -- 172.21.15.59:0/1694071241 learned_addr learned my addr 172.21.15.59:0/1694071241 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:14:47.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.655+0000 7fed6b7fe700 1 --2- 172.21.15.59:0/1694071241 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed6c0662e0 0x7fed6c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:47.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.655+0000 7fed6affd700 1 -- 172.21.15.59:0/1694071241 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed6c0662e0 msgr2=0x7fed6c0fd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:47.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.655+0000 7fed6affd700 1 --2- 172.21.15.59:0/1694071241 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed6c0662e0 0x7fed6c0fd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:47.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.656+0000 7fed6affd700 1 -- 172.21.15.59:0/1694071241 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed60005040 con 0x7fed6c0fff00 2024-03-19T03:14:47.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.656+0000 7fed6b7fe700 1 --2- 172.21.15.59:0/1694071241 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed6c0662e0 0x7fed6c0fd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:14:47.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.656+0000 7fed6affd700 1 --2- 172.21.15.59:0/1694071241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed6c0fff00 0x7fed6c0fb8c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fed60013870 tx=0x7fed60009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:47.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.656+0000 7fed53fff700 1 -- 172.21.15.59:0/1694071241 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed60021070 con 0x7fed6c0fff00 2024-03-19T03:14:47.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.656+0000 7fed71ceb700 1 -- 172.21.15.59:0/1694071241 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed6c0fc210 con 0x7fed6c0fff00 2024-03-19T03:14:47.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.656+0000 7fed71ceb700 1 -- 172.21.15.59:0/1694071241 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed6c115a70 con 0x7fed6c0fff00 2024-03-19T03:14:47.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.656+0000 7fed53fff700 1 -- 172.21.15.59:0/1694071241 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed6001c400 con 0x7fed6c0fff00 2024-03-19T03:14:47.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.656+0000 7fed53fff700 1 -- 172.21.15.59:0/1694071241 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed6000e070 con 0x7fed6c0fff00 2024-03-19T03:14:47.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.657+0000 7fed71ceb700 1 -- 172.21.15.59:0/1694071241 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed6c05f5f0 con 0x7fed6c0fff00 2024-03-19T03:14:47.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.659+0000 7fed53fff700 1 -- 172.21.15.59:0/1694071241 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fed6001ca20 con 0x7fed6c0fff00 2024-03-19T03:14:47.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.659+0000 7fed53fff700 1 --2- 172.21.15.59:0/1694071241 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fed540692a0 0x7fed5406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:47.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.659+0000 7fed53fff700 1 -- 172.21.15.59:0/1694071241 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fed60088540 con 0x7fed6c0fff00 2024-03-19T03:14:47.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.659+0000 7fed6b7fe700 1 --2- 172.21.15.59:0/1694071241 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fed540692a0 0x7fed5406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:47.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.660+0000 7fed6b7fe700 1 --2- 172.21.15.59:0/1694071241 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fed540692a0 0x7fed5406b750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fed6c0fd130 tx=0x7fed5c00b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:47.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.664+0000 7fed53fff700 1 -- 172.21.15.59:0/1694071241 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fed60054870 con 0x7fed6c0fff00 2024-03-19T03:14:47.701 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-03-19T03:14:47.704 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-19T03:14:47.704 DEBUG:teuthology.orchestra.run.smithi176:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e48b1f6c0608b2615e1890cc46286c3877d6e910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-03-19T03:14:47.763 INFO:tasks.workunit.client.1.smithi176.stderr:+ mkdir -p fsstress 2024-03-19T03:14:47.765 INFO:tasks.workunit.client.1.smithi176.stderr:+ pushd fsstress 2024-03-19T03:14:47.765 INFO:tasks.workunit.client.1.smithi176.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-19T03:14:47.765 INFO:tasks.workunit.client.1.smithi176.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-19T03:14:47.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.880+0000 7fed71ceb700 1 -- 172.21.15.59:0/1694071241 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fed6c0f8f70 con 0x7fed540692a0 2024-03-19T03:14:47.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.881+0000 7fed53fff700 1 -- 172.21.15.59:0/1694071241 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7fed6c0f8f70 con 0x7fed540692a0 2024-03-19T03:14:47.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.883+0000 7fed71ceb700 1 -- 172.21.15.59:0/1694071241 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fed540692a0 msgr2=0x7fed5406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:47.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.883+0000 7fed71ceb700 1 --2- 172.21.15.59:0/1694071241 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fed540692a0 0x7fed5406b750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fed6c0fd130 tx=0x7fed5c00b040 comp rx=0 tx=0).stop 2024-03-19T03:14:47.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.883+0000 7fed71ceb700 1 -- 172.21.15.59:0/1694071241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed6c0fff00 msgr2=0x7fed6c0fb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:47.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.883+0000 7fed71ceb700 1 --2- 172.21.15.59:0/1694071241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed6c0fff00 0x7fed6c0fb8c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fed60013870 tx=0x7fed60009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:47.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.884+0000 7fed71ceb700 1 -- 172.21.15.59:0/1694071241 shutdown_connections 2024-03-19T03:14:47.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.884+0000 7fed71ceb700 1 --2- 172.21.15.59:0/1694071241 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fed540692a0 0x7fed5406b750 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:47.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.884+0000 7fed71ceb700 1 --2- 172.21.15.59:0/1694071241 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed6c0662e0 0x7fed6c0fd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:47.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.884+0000 7fed71ceb700 1 --2- 172.21.15.59:0/1694071241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed6c0fff00 0x7fed6c0fb8c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:47.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.884+0000 7fed71ceb700 1 -- 172.21.15.59:0/1694071241 >> 172.21.15.59:0/1694071241 conn(0x7fed6c0f7430 msgr2=0x7fed6c0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:47.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.884+0000 7fed71ceb700 1 -- 172.21.15.59:0/1694071241 shutdown_connections 2024-03-19T03:14:47.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:47.884+0000 7fed71ceb700 1 -- 172.21.15.59:0/1694071241 wait complete. 2024-03-19T03:14:48.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.012+0000 7fc584f30700 1 -- 172.21.15.59:0/3177490227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5800fcf80 msgr2=0x7fc5800fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:48.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.012+0000 7fc584f30700 1 --2- 172.21.15.59:0/3177490227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5800fcf80 0x7fc5800fd3f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc574004940 tx=0x7fc574009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:48.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.012+0000 7fc584f30700 1 -- 172.21.15.59:0/3177490227 shutdown_connections 2024-03-19T03:14:48.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.012+0000 7fc584f30700 1 --2- 172.21.15.59:0/3177490227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5800fcf80 0x7fc5800fd3f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:48.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.012+0000 7fc584f30700 1 --2- 172.21.15.59:0/3177490227 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5800fbcd0 0x7fc5800fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:48.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.012+0000 7fc584f30700 1 -- 172.21.15.59:0/3177490227 >> 172.21.15.59:0/3177490227 conn(0x7fc5800f7430 msgr2=0x7fc5800f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:48.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.012+0000 7fc584f30700 1 -- 172.21.15.59:0/3177490227 shutdown_connections 2024-03-19T03:14:48.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.012+0000 7fc584f30700 1 -- 172.21.15.59:0/3177490227 wait complete. 2024-03-19T03:14:48.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.013+0000 7fc584f30700 1 Processor -- start 2024-03-19T03:14:48.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.013+0000 7fc584f30700 1 -- start start 2024-03-19T03:14:48.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.013+0000 7fc584f30700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5800fbcd0 0x7fc58010daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:48.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.013+0000 7fc584f30700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5800fcf80 0x7fc58010dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:48.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.013+0000 7fc584f30700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58010e5e0 con 0x7fc5800fcf80 2024-03-19T03:14:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.013+0000 7fc584f30700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58010ed90 con 0x7fc5800fbcd0 2024-03-19T03:14:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.013+0000 7fc57dd9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5800fcf80 0x7fc58010dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.013+0000 7fc57e59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5800fbcd0 0x7fc58010daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.013+0000 7fc57dd9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5800fcf80 0x7fc58010dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60470/0 (socket says 172.21.15.59:60470) 2024-03-19T03:14:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.013+0000 7fc57dd9b700 1 -- 172.21.15.59:0/1077567669 learned_addr learned my addr 172.21.15.59:0/1077567669 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:14:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.013+0000 7fc57dd9b700 1 -- 172.21.15.59:0/1077567669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5800fbcd0 msgr2=0x7fc58010daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.013+0000 7fc57dd9b700 1 --2- 172.21.15.59:0/1077567669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5800fbcd0 0x7fc58010daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.013+0000 7fc57dd9b700 1 -- 172.21.15.59:0/1077567669 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc574005040 con 0x7fc5800fcf80 2024-03-19T03:14:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.013+0000 7fc57e59c700 1 --2- 172.21.15.59:0/1077567669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5800fbcd0 0x7fc58010daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:14:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.014+0000 7fc57dd9b700 1 --2- 172.21.15.59:0/1077567669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5800fcf80 0x7fc58010dfe0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc574009150 tx=0x7fc574009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.014+0000 7fc56effd700 1 -- 172.21.15.59:0/1077567669 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc57400e040 con 0x7fc5800fcf80 2024-03-19T03:14:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.014+0000 7fc56effd700 1 -- 172.21.15.59:0/1077567669 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc574004290 con 0x7fc5800fcf80 2024-03-19T03:14:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.014+0000 7fc584f30700 1 -- 172.21.15.59:0/1077567669 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc58010f010 con 0x7fc5800fcf80 2024-03-19T03:14:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.014+0000 7fc56effd700 1 -- 172.21.15.59:0/1077567669 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc57401c630 con 0x7fc5800fcf80 2024-03-19T03:14:48.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.014+0000 7fc584f30700 1 -- 172.21.15.59:0/1077567669 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc58010f550 con 0x7fc5800fcf80 2024-03-19T03:14:48.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.015+0000 7fc584f30700 1 -- 172.21.15.59:0/1077567669 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc58005f5f0 con 0x7fc5800fcf80 2024-03-19T03:14:48.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.016+0000 7fc56effd700 1 -- 172.21.15.59:0/1077567669 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc57400bbf0 con 0x7fc5800fcf80 2024-03-19T03:14:48.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.016+0000 7fc56effd700 1 --2- 172.21.15.59:0/1077567669 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc5680692a0 0x7fc56806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:48.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.016+0000 7fc56effd700 1 -- 172.21.15.59:0/1077567669 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fc574087e60 con 0x7fc5800fcf80 2024-03-19T03:14:48.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.016+0000 7fc57e59c700 1 --2- 172.21.15.59:0/1077567669 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc5680692a0 0x7fc56806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:48.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.017+0000 7fc57e59c700 1 --2- 172.21.15.59:0/1077567669 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc5680692a0 0x7fc56806b750 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc57000f7c0 tx=0x7fc570011040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:48.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.021+0000 7fc56effd700 1 -- 172.21.15.59:0/1077567669 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fc574057c20 con 0x7fc5800fcf80 2024-03-19T03:14:48.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.227+0000 7fc584f30700 1 -- 172.21.15.59:0/1077567669 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc58010f830 con 0x7fc5680692a0 2024-03-19T03:14:48.228 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:48 smithi059 ceph-mon[29133]: pgmap v221: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-19T03:14:48.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.240+0000 7fc56effd700 1 -- 172.21.15.59:0/1077567669 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7fc58010f830 con 0x7fc5680692a0 2024-03-19T03:14:48.242 INFO:teuthology.orchestra.run.smithi059.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-19T03:14:48.242 INFO:teuthology.orchestra.run.smithi059.stdout:alertmanager.smithi059 smithi059 *:9093,9094 running (5m) 63s ago 6m 27.3M - 0.25.0 c8568f914cd2 40a81ac05a90 2024-03-19T03:14:48.242 INFO:teuthology.orchestra.run.smithi059.stdout:crash.smithi059 smithi059 running (7m) 63s ago 7m 6669k - 17.2.7-700-gb4843bd7 810955603526 3090e46c4bff 2024-03-19T03:14:48.242 INFO:teuthology.orchestra.run.smithi059.stdout:crash.smithi176 smithi176 running (6m) 69s ago 6m 6660k - 17.2.7-700-gb4843bd7 810955603526 2df9a0e61e4c 2024-03-19T03:14:48.242 INFO:teuthology.orchestra.run.smithi059.stdout:grafana.smithi059 smithi059 *:3000 running (5m) 63s ago 6m 85.4M - 9.4.7 954c08fa6188 b3394e3d4026 2024-03-19T03:14:48.242 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi059.doeuxg smithi059 running (82s) 63s ago 81s 14.5M - 17.2.7-700-gb4843bd7 810955603526 18d717af1972 2024-03-19T03:14:48.242 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi059.hmylwa smithi059 running (87s) 63s ago 87s 14.6M - 17.2.7-700-gb4843bd7 810955603526 d4acd92451b4 2024-03-19T03:14:48.243 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi176.bochhq smithi176 running (85s) 69s ago 84s 14.3M - 17.2.7-700-gb4843bd7 810955603526 05e9bc441018 2024-03-19T03:14:48.243 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi176.gisnqc smithi176 running (79s) 69s ago 78s 12.2M - 17.2.7-700-gb4843bd7 810955603526 75982373b447 2024-03-19T03:14:48.243 INFO:teuthology.orchestra.run.smithi059.stdout:mgr.smithi059.azpecc smithi059 *:9283 running (8m) 63s ago 8m 480M - 17.2.7-700-gb4843bd7 810955603526 441ca0cfcfac 2024-03-19T03:14:48.243 INFO:teuthology.orchestra.run.smithi059.stdout:mgr.smithi176.ycmqbw smithi176 *:8443,9283 running (5m) 69s ago 5m 433M - 17.2.7-700-gb4843bd7 810955603526 ac8bc9c6ec9b 2024-03-19T03:14:48.243 INFO:teuthology.orchestra.run.smithi059.stdout:mon.smithi059 smithi059 running (8m) 63s ago 8m 49.9M 2048M 17.2.7-700-gb4843bd7 810955603526 c8bc018eaa37 2024-03-19T03:14:48.243 INFO:teuthology.orchestra.run.smithi059.stdout:mon.smithi176 smithi176 running (5m) 69s ago 5m 33.4M 2048M 17.2.7-700-gb4843bd7 810955603526 4130cce6243b 2024-03-19T03:14:48.243 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter.smithi059 smithi059 *:9100 running (7m) 63s ago 7m 16.1M - 1.5.0 0da6a335fe13 d3fca5e9a667 2024-03-19T03:14:48.243 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter.smithi176 smithi176 *:9100 running (6m) 69s ago 6m 14.7M - 1.5.0 0da6a335fe13 35389a3c33fd 2024-03-19T03:14:48.243 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 smithi059 running (5m) 63s ago 5m 46.3M 4289M 17.2.7-700-gb4843bd7 810955603526 2511279e8852 2024-03-19T03:14:48.243 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 smithi059 running (4m) 63s ago 4m 45.0M 4289M 17.2.7-700-gb4843bd7 810955603526 cd85f348d650 2024-03-19T03:14:48.243 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 smithi059 running (4m) 63s ago 4m 47.0M 4289M 17.2.7-700-gb4843bd7 810955603526 363582327fde 2024-03-19T03:14:48.243 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 smithi176 running (3m) 69s ago 3m 43.5M 5312M 17.2.7-700-gb4843bd7 810955603526 436c587b5265 2024-03-19T03:14:48.243 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 smithi176 running (3m) 69s ago 3m 43.1M 5312M 17.2.7-700-gb4843bd7 810955603526 80d2dd2cd18b 2024-03-19T03:14:48.244 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 smithi176 running (2m) 69s ago 2m 41.6M 5312M 17.2.7-700-gb4843bd7 810955603526 dd0f98b5ec05 2024-03-19T03:14:48.244 INFO:teuthology.orchestra.run.smithi059.stdout:prometheus.smithi059 smithi059 *:9095 running (5m) 63s ago 6m 54.1M - 2.43.0 a07b618ecd1d 1782702019d3 2024-03-19T03:14:48.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.242+0000 7fc584f30700 1 -- 172.21.15.59:0/1077567669 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc5680692a0 msgr2=0x7fc56806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:48.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.242+0000 7fc584f30700 1 --2- 172.21.15.59:0/1077567669 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc5680692a0 0x7fc56806b750 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc57000f7c0 tx=0x7fc570011040 comp rx=0 tx=0).stop 2024-03-19T03:14:48.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.242+0000 7fc584f30700 1 -- 172.21.15.59:0/1077567669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5800fcf80 msgr2=0x7fc58010dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:48.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.242+0000 7fc584f30700 1 --2- 172.21.15.59:0/1077567669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5800fcf80 0x7fc58010dfe0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc574009150 tx=0x7fc574009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:48.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.243+0000 7fc584f30700 1 -- 172.21.15.59:0/1077567669 shutdown_connections 2024-03-19T03:14:48.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.243+0000 7fc584f30700 1 --2- 172.21.15.59:0/1077567669 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc5680692a0 0x7fc56806b750 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:48.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.243+0000 7fc584f30700 1 --2- 172.21.15.59:0/1077567669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5800fbcd0 0x7fc58010daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:48.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.243+0000 7fc584f30700 1 --2- 172.21.15.59:0/1077567669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5800fcf80 0x7fc58010dfe0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:48.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.243+0000 7fc584f30700 1 -- 172.21.15.59:0/1077567669 >> 172.21.15.59:0/1077567669 conn(0x7fc5800f7430 msgr2=0x7fc580065b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:48.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.243+0000 7fc584f30700 1 -- 172.21.15.59:0/1077567669 shutdown_connections 2024-03-19T03:14:48.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.243+0000 7fc584f30700 1 -- 172.21.15.59:0/1077567669 wait complete. 2024-03-19T03:14:48.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.347+0000 7fd03dfcb700 1 -- 172.21.15.59:0/247713847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd038066ad0 msgr2=0x7fd0380ffa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:48.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.347+0000 7fd03dfcb700 1 --2- 172.21.15.59:0/247713847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd038066ad0 0x7fd0380ffa70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd028004940 tx=0x7fd028009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:48.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.347+0000 7fd03dfcb700 1 -- 172.21.15.59:0/247713847 shutdown_connections 2024-03-19T03:14:48.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.347+0000 7fd03dfcb700 1 --2- 172.21.15.59:0/247713847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0380fffb0 0x7fd038102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:48.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.347+0000 7fd03dfcb700 1 --2- 172.21.15.59:0/247713847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd038066ad0 0x7fd0380ffa70 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:48.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.347+0000 7fd03dfcb700 1 -- 172.21.15.59:0/247713847 >> 172.21.15.59:0/247713847 conn(0x7fd0380f7480 msgr2=0x7fd0380f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:48.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.347+0000 7fd03dfcb700 1 -- 172.21.15.59:0/247713847 shutdown_connections 2024-03-19T03:14:48.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.347+0000 7fd03dfcb700 1 -- 172.21.15.59:0/247713847 wait complete. 2024-03-19T03:14:48.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.347+0000 7fd03dfcb700 1 Processor -- start 2024-03-19T03:14:48.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.348+0000 7fd03dfcb700 1 -- start start 2024-03-19T03:14:48.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.348+0000 7fd03dfcb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0380fffb0 0x7fd03810db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:48.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.348+0000 7fd03dfcb700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd03810e0c0 0x7fd03810ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:48.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.348+0000 7fd03dfcb700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03810e580 con 0x7fd0380fffb0 2024-03-19T03:14:48.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.348+0000 7fd03dfcb700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03810e6c0 con 0x7fd03810e0c0 2024-03-19T03:14:48.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.348+0000 7fd0377fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0380fffb0 0x7fd03810db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:48.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.348+0000 7fd0377fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0380fffb0 0x7fd03810db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60488/0 (socket says 172.21.15.59:60488) 2024-03-19T03:14:48.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.348+0000 7fd0377fe700 1 -- 172.21.15.59:0/2999131765 learned_addr learned my addr 172.21.15.59:0/2999131765 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:14:48.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.348+0000 7fd036ffd700 1 --2- 172.21.15.59:0/2999131765 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd03810e0c0 0x7fd03810ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:48.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.349+0000 7fd0377fe700 1 -- 172.21.15.59:0/2999131765 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd03810e0c0 msgr2=0x7fd03810ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:48.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.349+0000 7fd0377fe700 1 --2- 172.21.15.59:0/2999131765 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd03810e0c0 0x7fd03810ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:48.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.349+0000 7fd0377fe700 1 -- 172.21.15.59:0/2999131765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd028005040 con 0x7fd0380fffb0 2024-03-19T03:14:48.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.349+0000 7fd036ffd700 1 --2- 172.21.15.59:0/2999131765 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd03810e0c0 0x7fd03810ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:14:48.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.349+0000 7fd0377fe700 1 --2- 172.21.15.59:0/2999131765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0380fffb0 0x7fd03810db80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd028013870 tx=0x7fd028009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:48.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.349+0000 7fd01ffff700 1 -- 172.21.15.59:0/2999131765 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd02800e030 con 0x7fd0380fffb0 2024-03-19T03:14:48.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.349+0000 7fd01ffff700 1 -- 172.21.15.59:0/2999131765 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd028013de0 con 0x7fd0380fffb0 2024-03-19T03:14:48.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.349+0000 7fd03dfcb700 1 -- 172.21.15.59:0/2999131765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd03810f510 con 0x7fd0380fffb0 2024-03-19T03:14:48.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.349+0000 7fd03dfcb700 1 -- 172.21.15.59:0/2999131765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd03810fa20 con 0x7fd0380fffb0 2024-03-19T03:14:48.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.349+0000 7fd01ffff700 1 -- 172.21.15.59:0/2999131765 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd02800b410 con 0x7fd0380fffb0 2024-03-19T03:14:48.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.350+0000 7fd03dfcb700 1 -- 172.21.15.59:0/2999131765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd03805f5f0 con 0x7fd0380fffb0 2024-03-19T03:14:48.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.351+0000 7fd01ffff700 1 -- 172.21.15.59:0/2999131765 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd02801c770 con 0x7fd0380fffb0 2024-03-19T03:14:48.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.352+0000 7fd01ffff700 1 --2- 172.21.15.59:0/2999131765 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd0200692a0 0x7fd02006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:48.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.352+0000 7fd01ffff700 1 -- 172.21.15.59:0/2999131765 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fd028088870 con 0x7fd0380fffb0 2024-03-19T03:14:48.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.352+0000 7fd036ffd700 1 --2- 172.21.15.59:0/2999131765 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd0200692a0 0x7fd02006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:48.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.352+0000 7fd036ffd700 1 --2- 172.21.15.59:0/2999131765 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd0200692a0 0x7fd02006b750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd02c009b80 tx=0x7fd02c00b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:48.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.359+0000 7fd01ffff700 1 -- 172.21.15.59:0/2999131765 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fd028058630 con 0x7fd0380fffb0 2024-03-19T03:14:48.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:48 smithi176 ceph-mon[36450]: pgmap v221: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-03-19T03:14:48.429 INFO:tasks.workunit.client.1.smithi176.stderr:+ tar xzf ltp-full.tgz 2024-03-19T03:14:48.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.820+0000 7fd03dfcb700 1 -- 172.21.15.59:0/2999131765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd0380008d0 con 0x7fd0380fffb0 2024-03-19T03:14:48.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.821+0000 7fd01ffff700 1 -- 172.21.15.59:0/2999131765 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7fd02801c400 con 0x7fd0380fffb0 2024-03-19T03:14:48.823 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-19T03:14:48.823 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": { 2024-03-19T03:14:48.823 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-700-gb4843bd7 (b4843bd759866f9ee1bdfb1dfa084ec7d085b910) quincy (stable)": 2 2024-03-19T03:14:48.823 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-19T03:14:48.823 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": { 2024-03-19T03:14:48.824 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-700-gb4843bd7 (b4843bd759866f9ee1bdfb1dfa084ec7d085b910) quincy (stable)": 2 2024-03-19T03:14:48.824 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-19T03:14:48.824 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": { 2024-03-19T03:14:48.824 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-700-gb4843bd7 (b4843bd759866f9ee1bdfb1dfa084ec7d085b910) quincy (stable)": 6 2024-03-19T03:14:48.824 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-19T03:14:48.824 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": { 2024-03-19T03:14:48.824 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-700-gb4843bd7 (b4843bd759866f9ee1bdfb1dfa084ec7d085b910) quincy (stable)": 4 2024-03-19T03:14:48.825 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-19T03:14:48.825 INFO:teuthology.orchestra.run.smithi059.stdout: "overall": { 2024-03-19T03:14:48.825 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-700-gb4843bd7 (b4843bd759866f9ee1bdfb1dfa084ec7d085b910) quincy (stable)": 14 2024-03-19T03:14:48.825 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-19T03:14:48.825 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-19T03:14:48.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.824+0000 7fd03dfcb700 1 -- 172.21.15.59:0/2999131765 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd0200692a0 msgr2=0x7fd02006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:48.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.824+0000 7fd03dfcb700 1 --2- 172.21.15.59:0/2999131765 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd0200692a0 0x7fd02006b750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd02c009b80 tx=0x7fd02c00b040 comp rx=0 tx=0).stop 2024-03-19T03:14:48.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.824+0000 7fd03dfcb700 1 -- 172.21.15.59:0/2999131765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0380fffb0 msgr2=0x7fd03810db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:48.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.824+0000 7fd03dfcb700 1 --2- 172.21.15.59:0/2999131765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0380fffb0 0x7fd03810db80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd028013870 tx=0x7fd028009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:48.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.824+0000 7fd03dfcb700 1 -- 172.21.15.59:0/2999131765 shutdown_connections 2024-03-19T03:14:48.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.824+0000 7fd03dfcb700 1 --2- 172.21.15.59:0/2999131765 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fd0200692a0 0x7fd02006b750 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:48.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.824+0000 7fd03dfcb700 1 --2- 172.21.15.59:0/2999131765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0380fffb0 0x7fd03810db80 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:48.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.824+0000 7fd03dfcb700 1 --2- 172.21.15.59:0/2999131765 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd03810e0c0 0x7fd03810ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:48.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.824+0000 7fd03dfcb700 1 -- 172.21.15.59:0/2999131765 >> 172.21.15.59:0/2999131765 conn(0x7fd0380f7480 msgr2=0x7fd0380f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:48.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.824+0000 7fd03dfcb700 1 -- 172.21.15.59:0/2999131765 shutdown_connections 2024-03-19T03:14:48.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.825+0000 7fd03dfcb700 1 -- 172.21.15.59:0/2999131765 wait complete. 2024-03-19T03:14:48.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.985+0000 7f2f1aa50700 1 -- 172.21.15.59:0/3705931877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f14102380 msgr2=0x7f2f14104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:48.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.985+0000 7f2f1aa50700 1 --2- 172.21.15.59:0/3705931877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f14102380 0x7f2f14104760 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2f08004880 tx=0x7f2f08009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:48.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.985+0000 7f2f1aa50700 1 -- 172.21.15.59:0/3705931877 shutdown_connections 2024-03-19T03:14:48.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.985+0000 7f2f1aa50700 1 --2- 172.21.15.59:0/3705931877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f14102380 0x7f2f14104760 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:48.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.985+0000 7f2f1aa50700 1 --2- 172.21.15.59:0/3705931877 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f14066260 0x7f2f14101e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:48.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.985+0000 7f2f1aa50700 1 -- 172.21.15.59:0/3705931877 >> 172.21.15.59:0/3705931877 conn(0x7f2f140f7430 msgr2=0x7f2f140f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:48.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.985+0000 7f2f1aa50700 1 -- 172.21.15.59:0/3705931877 shutdown_connections 2024-03-19T03:14:48.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.985+0000 7f2f1aa50700 1 -- 172.21.15.59:0/3705931877 wait complete. 2024-03-19T03:14:48.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.986+0000 7f2f1aa50700 1 Processor -- start 2024-03-19T03:14:48.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.986+0000 7f2f1aa50700 1 -- start start 2024-03-19T03:14:48.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.987+0000 7f2f1aa50700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f14066260 0x7f2f1410fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:48.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.987+0000 7f2f1aa50700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f14102380 0x7f2f141102c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:48.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.987+0000 7f2f1aa50700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f141108c0 con 0x7f2f14066260 2024-03-19T03:14:48.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.987+0000 7f2f1aa50700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f14110fb0 con 0x7f2f14102380 2024-03-19T03:14:48.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.987+0000 7f2f13fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f14066260 0x7f2f1410fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:48.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.987+0000 7f2f137fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f14102380 0x7f2f141102c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:48.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.987+0000 7f2f137fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f14102380 0x7f2f141102c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.59:36180/0 (socket says 172.21.15.59:36180) 2024-03-19T03:14:48.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.987+0000 7f2f137fe700 1 -- 172.21.15.59:0/1999324382 learned_addr learned my addr 172.21.15.59:0/1999324382 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:14:48.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.987+0000 7f2f137fe700 1 -- 172.21.15.59:0/1999324382 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f14066260 msgr2=0x7f2f1410fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:48.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.987+0000 7f2f137fe700 1 --2- 172.21.15.59:0/1999324382 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f14066260 0x7f2f1410fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:48.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.987+0000 7f2f137fe700 1 -- 172.21.15.59:0/1999324382 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f04005040 con 0x7f2f14102380 2024-03-19T03:14:48.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.988+0000 7f2f137fe700 1 --2- 172.21.15.59:0/1999324382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f14102380 0x7f2f141102c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2f08000c00 tx=0x7f2f08009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:48.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.988+0000 7f2f10ff9700 1 -- 172.21.15.59:0/1999324382 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f0800e030 con 0x7f2f14102380 2024-03-19T03:14:48.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.988+0000 7f2f1aa50700 1 -- 172.21.15.59:0/1999324382 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f08005040 con 0x7f2f14102380 2024-03-19T03:14:48.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.988+0000 7f2f10ff9700 1 -- 172.21.15.59:0/1999324382 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f08013e90 con 0x7f2f14102380 2024-03-19T03:14:48.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.988+0000 7f2f10ff9700 1 -- 172.21.15.59:0/1999324382 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f0801cdf0 con 0x7f2f14102380 2024-03-19T03:14:48.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.988+0000 7f2f1aa50700 1 -- 172.21.15.59:0/1999324382 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f14111660 con 0x7f2f14102380 2024-03-19T03:14:48.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.989+0000 7f2f1aa50700 1 -- 172.21.15.59:0/1999324382 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f1405f5f0 con 0x7f2f14102380 2024-03-19T03:14:48.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.990+0000 7f2f10ff9700 1 -- 172.21.15.59:0/1999324382 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f2f080139c0 con 0x7f2f14102380 2024-03-19T03:14:48.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.991+0000 7f2f10ff9700 1 --2- 172.21.15.59:0/1999324382 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2efc0692a0 0x7f2efc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:48.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.991+0000 7f2f13fff700 1 --2- 172.21.15.59:0/1999324382 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2efc0692a0 0x7f2efc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:48.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.991+0000 7f2f10ff9700 1 -- 172.21.15.59:0/1999324382 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f2f08088ad0 con 0x7f2f14102380 2024-03-19T03:14:48.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.991+0000 7f2f13fff700 1 --2- 172.21.15.59:0/1999324382 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2efc0692a0 0x7f2efc06b750 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2f14065fa0 tx=0x7f2f0400b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:48.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:48.995+0000 7f2f10ff9700 1 -- 172.21.15.59:0/1999324382 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f2f080588b0 con 0x7f2f14102380 2024-03-19T03:14:49.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:49 smithi059 ceph-mon[29133]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:14:49.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:49 smithi059 ceph-mon[29133]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:14:49.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:49 smithi059 ceph-mon[29133]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:14:49.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:49 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/2999131765' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-19T03:14:49.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.315+0000 7f2f1aa50700 1 -- 172.21.15.59:0/1999324382 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2f14111910 con 0x7f2f14102380 2024-03-19T03:14:49.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.316+0000 7f2f10ff9700 1 -- 172.21.15.59:0/1999324382 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1806 (secure 0 0 0) 0x7f2f08058440 con 0x7f2f14102380 2024-03-19T03:14:49.318 INFO:teuthology.orchestra.run.smithi059.stdout:e14 2024-03-19T03:14:49.318 INFO:teuthology.orchestra.run.smithi059.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-19T03:14:49.318 INFO:teuthology.orchestra.run.smithi059.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-19T03:14:49.318 INFO:teuthology.orchestra.run.smithi059.stdout:legacy client fscid: 1 2024-03-19T03:14:49.318 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:14:49.318 INFO:teuthology.orchestra.run.smithi059.stdout:Filesystem 'cephfs' (1) 2024-03-19T03:14:49.318 INFO:teuthology.orchestra.run.smithi059.stdout:fs_name cephfs 2024-03-19T03:14:49.318 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 13 2024-03-19T03:14:49.318 INFO:teuthology.orchestra.run.smithi059.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-19T03:14:49.318 INFO:teuthology.orchestra.run.smithi059.stdout:created 2024-03-19T03:13:17.039133+0000 2024-03-19T03:14:49.318 INFO:teuthology.orchestra.run.smithi059.stdout:modified 2024-03-19T03:13:32.433876+0000 2024-03-19T03:14:49.319 INFO:teuthology.orchestra.run.smithi059.stdout:tableserver 0 2024-03-19T03:14:49.319 INFO:teuthology.orchestra.run.smithi059.stdout:root 0 2024-03-19T03:14:49.319 INFO:teuthology.orchestra.run.smithi059.stdout:session_timeout 60 2024-03-19T03:14:49.319 INFO:teuthology.orchestra.run.smithi059.stdout:session_autoclose 300 2024-03-19T03:14:49.319 INFO:teuthology.orchestra.run.smithi059.stdout:max_file_size 1099511627776 2024-03-19T03:14:49.319 INFO:teuthology.orchestra.run.smithi059.stdout:required_client_features {} 2024-03-19T03:14:49.319 INFO:teuthology.orchestra.run.smithi059.stdout:last_failure 0 2024-03-19T03:14:49.319 INFO:teuthology.orchestra.run.smithi059.stdout:last_failure_osd_epoch 0 2024-03-19T03:14:49.319 INFO:teuthology.orchestra.run.smithi059.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-19T03:14:49.319 INFO:teuthology.orchestra.run.smithi059.stdout:max_mds 1 2024-03-19T03:14:49.319 INFO:teuthology.orchestra.run.smithi059.stdout:in 0 2024-03-19T03:14:49.319 INFO:teuthology.orchestra.run.smithi059.stdout:up {0=14468} 2024-03-19T03:14:49.319 INFO:teuthology.orchestra.run.smithi059.stdout:failed 2024-03-19T03:14:49.319 INFO:teuthology.orchestra.run.smithi059.stdout:damaged 2024-03-19T03:14:49.319 INFO:teuthology.orchestra.run.smithi059.stdout:stopped 2024-03-19T03:14:49.320 INFO:teuthology.orchestra.run.smithi059.stdout:data_pools [3] 2024-03-19T03:14:49.320 INFO:teuthology.orchestra.run.smithi059.stdout:metadata_pool 2 2024-03-19T03:14:49.320 INFO:teuthology.orchestra.run.smithi059.stdout:inline_data enabled 2024-03-19T03:14:49.320 INFO:teuthology.orchestra.run.smithi059.stdout:balancer 2024-03-19T03:14:49.320 INFO:teuthology.orchestra.run.smithi059.stdout:standby_count_wanted 1 2024-03-19T03:14:49.320 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi059.hmylwa{0:14468} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.59:6826/2936511711,v1:172.21.15.59:6827/2936511711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-19T03:14:49.320 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:14:49.320 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:14:49.320 INFO:teuthology.orchestra.run.smithi059.stdout:Standby daemons: 2024-03-19T03:14:49.320 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-19T03:14:49.320 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi059.doeuxg{-1:14482} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.59:6828/503726080,v1:172.21.15.59:6829/503726080] compat {c=[1],r=[1],i=[7ff]}] 2024-03-19T03:14:49.320 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi176.bochhq{-1:24279} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.176:6824/193753714,v1:172.21.15.176:6825/193753714] compat {c=[1],r=[1],i=[7ff]}] 2024-03-19T03:14:49.320 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi176.gisnqc{-1:24289} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.176:6826/4058958323,v1:172.21.15.176:6827/4058958323] compat {c=[1],r=[1],i=[7ff]}] 2024-03-19T03:14:49.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.319+0000 7f2f1aa50700 1 -- 172.21.15.59:0/1999324382 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2efc0692a0 msgr2=0x7f2efc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:49.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.319+0000 7f2f1aa50700 1 --2- 172.21.15.59:0/1999324382 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2efc0692a0 0x7f2efc06b750 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2f14065fa0 tx=0x7f2f0400b040 comp rx=0 tx=0).stop 2024-03-19T03:14:49.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.319+0000 7f2f1aa50700 1 -- 172.21.15.59:0/1999324382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f14102380 msgr2=0x7f2f141102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:49.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.319+0000 7f2f1aa50700 1 --2- 172.21.15.59:0/1999324382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f14102380 0x7f2f141102c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2f08000c00 tx=0x7f2f08009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:49.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.319+0000 7f2f1aa50700 1 -- 172.21.15.59:0/1999324382 shutdown_connections 2024-03-19T03:14:49.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.319+0000 7f2f1aa50700 1 --2- 172.21.15.59:0/1999324382 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2efc0692a0 0x7f2efc06b750 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:49.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.319+0000 7f2f1aa50700 1 --2- 172.21.15.59:0/1999324382 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f14066260 0x7f2f1410fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:49.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.319+0000 7f2f1aa50700 1 --2- 172.21.15.59:0/1999324382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f14102380 0x7f2f141102c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:49.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.319+0000 7f2f1aa50700 1 -- 172.21.15.59:0/1999324382 >> 172.21.15.59:0/1999324382 conn(0x7f2f140f7430 msgr2=0x7f2f140f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:49.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.319+0000 7f2f1aa50700 1 -- 172.21.15.59:0/1999324382 shutdown_connections 2024-03-19T03:14:49.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.319+0000 7f2f1aa50700 1 -- 172.21.15.59:0/1999324382 wait complete. 2024-03-19T03:14:49.322 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 14 2024-03-19T03:14:49.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:49 smithi176 ceph-mon[36450]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:14:49.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:49 smithi176 ceph-mon[36450]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:14:49.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:49 smithi176 ceph-mon[36450]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:14:49.378 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:49 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/2999131765' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-19T03:14:49.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.442+0000 7fe1a73a4700 1 -- 172.21.15.59:0/3630413513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a00fbcd0 msgr2=0x7fe1a00fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:49.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.442+0000 7fe1a73a4700 1 --2- 172.21.15.59:0/3630413513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a00fbcd0 0x7fe1a00fc0e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe19c004940 tx=0x7fe19c009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:49.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.442+0000 7fe1a73a4700 1 -- 172.21.15.59:0/3630413513 shutdown_connections 2024-03-19T03:14:49.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.442+0000 7fe1a73a4700 1 --2- 172.21.15.59:0/3630413513 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1a00fcf80 0x7fe1a00fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:49.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.442+0000 7fe1a73a4700 1 --2- 172.21.15.59:0/3630413513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a00fbcd0 0x7fe1a00fc0e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:49.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.442+0000 7fe1a73a4700 1 -- 172.21.15.59:0/3630413513 >> 172.21.15.59:0/3630413513 conn(0x7fe1a00f7430 msgr2=0x7fe1a00f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:49.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.442+0000 7fe1a73a4700 1 -- 172.21.15.59:0/3630413513 shutdown_connections 2024-03-19T03:14:49.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.442+0000 7fe1a73a4700 1 -- 172.21.15.59:0/3630413513 wait complete. 2024-03-19T03:14:49.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.443+0000 7fe1a73a4700 1 Processor -- start 2024-03-19T03:14:49.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.443+0000 7fe1a73a4700 1 -- start start 2024-03-19T03:14:49.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.443+0000 7fe1a73a4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a00fbcd0 0x7fe1a010d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:49.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.443+0000 7fe1a73a4700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1a00fcf80 0x7fe1a010df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:49.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.443+0000 7fe1a73a4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a010e500 con 0x7fe1a00fbcd0 2024-03-19T03:14:49.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.443+0000 7fe1a73a4700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a010ed00 con 0x7fe1a00fcf80 2024-03-19T03:14:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.443+0000 7fe1a5140700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a00fbcd0 0x7fe1a010d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.443+0000 7fe1a5140700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a00fbcd0 0x7fe1a010d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60536/0 (socket says 172.21.15.59:60536) 2024-03-19T03:14:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.443+0000 7fe1a5140700 1 -- 172.21.15.59:0/266939926 learned_addr learned my addr 172.21.15.59:0/266939926 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:14:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.443+0000 7fe1a493f700 1 --2- 172.21.15.59:0/266939926 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1a00fcf80 0x7fe1a010df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.444+0000 7fe1a493f700 1 -- 172.21.15.59:0/266939926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a00fbcd0 msgr2=0x7fe1a010d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.444+0000 7fe1a493f700 1 --2- 172.21.15.59:0/266939926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a00fbcd0 0x7fe1a010d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.444+0000 7fe1a493f700 1 -- 172.21.15.59:0/266939926 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c005040 con 0x7fe1a00fcf80 2024-03-19T03:14:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.444+0000 7fe1a5140700 1 --2- 172.21.15.59:0/266939926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a00fbcd0 0x7fe1a010d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:14:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.444+0000 7fe1a493f700 1 --2- 172.21.15.59:0/266939926 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1a00fcf80 0x7fe1a010df00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe190009fd0 tx=0x7fe19000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.444+0000 7fe195ffb700 1 -- 172.21.15.59:0/266939926 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe190004e90 con 0x7fe1a00fcf80 2024-03-19T03:14:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.444+0000 7fe195ffb700 1 -- 172.21.15.59:0/266939926 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe19000f040 con 0x7fe1a00fcf80 2024-03-19T03:14:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.444+0000 7fe195ffb700 1 -- 172.21.15.59:0/266939926 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe19000ca50 con 0x7fe1a00fcf80 2024-03-19T03:14:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.444+0000 7fe1a73a4700 1 -- 172.21.15.59:0/266939926 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a010ef80 con 0x7fe1a00fcf80 2024-03-19T03:14:49.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.445+0000 7fe1a73a4700 1 -- 172.21.15.59:0/266939926 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a010f3b0 con 0x7fe1a00fcf80 2024-03-19T03:14:49.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.446+0000 7fe195ffb700 1 -- 172.21.15.59:0/266939926 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe190007cc0 con 0x7fe1a00fcf80 2024-03-19T03:14:49.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.446+0000 7fe1a73a4700 1 -- 172.21.15.59:0/266939926 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a0101b10 con 0x7fe1a00fcf80 2024-03-19T03:14:49.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.446+0000 7fe195ffb700 1 --2- 172.21.15.59:0/266939926 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe18c0692a0 0x7fe18c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:49.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.446+0000 7fe195ffb700 1 -- 172.21.15.59:0/266939926 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fe190086c90 con 0x7fe1a00fcf80 2024-03-19T03:14:49.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.446+0000 7fe1a5140700 1 --2- 172.21.15.59:0/266939926 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe18c0692a0 0x7fe18c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:49.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.447+0000 7fe1a5140700 1 --2- 172.21.15.59:0/266939926 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe18c0692a0 0x7fe18c06b750 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe19c00b0d0 tx=0x7fe19c009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:49.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.451+0000 7fe195ffb700 1 -- 172.21.15.59:0/266939926 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fe19004de50 con 0x7fe1a00fcf80 2024-03-19T03:14:49.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.694+0000 7fe1a73a4700 1 -- 172.21.15.59:0/266939926 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1a0104930 con 0x7fe18c0692a0 2024-03-19T03:14:50.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.696+0000 7fe195ffb700 1 -- 172.21.15.59:0/266939926 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7fe1a0104930 con 0x7fe18c0692a0 2024-03-19T03:14:50.012 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-19T03:14:50.012 INFO:teuthology.orchestra.run.smithi059.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639", 2024-03-19T03:14:50.012 INFO:teuthology.orchestra.run.smithi059.stdout: "in_progress": true, 2024-03-19T03:14:50.013 INFO:teuthology.orchestra.run.smithi059.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-19T03:14:50.013 INFO:teuthology.orchestra.run.smithi059.stdout: "services_complete": [], 2024-03-19T03:14:50.013 INFO:teuthology.orchestra.run.smithi059.stdout: "progress": "0/21 daemons upgraded", 2024-03-19T03:14:50.013 INFO:teuthology.orchestra.run.smithi059.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 image on host smithi176", 2024-03-19T03:14:50.013 INFO:teuthology.orchestra.run.smithi059.stdout: "is_paused": false 2024-03-19T03:14:50.013 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-19T03:14:50.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.698+0000 7fe1a73a4700 1 -- 172.21.15.59:0/266939926 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe18c0692a0 msgr2=0x7fe18c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:50.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.698+0000 7fe1a73a4700 1 --2- 172.21.15.59:0/266939926 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe18c0692a0 0x7fe18c06b750 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe19c00b0d0 tx=0x7fe19c009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:50.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.698+0000 7fe1a73a4700 1 -- 172.21.15.59:0/266939926 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1a00fcf80 msgr2=0x7fe1a010df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:50.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.698+0000 7fe1a73a4700 1 --2- 172.21.15.59:0/266939926 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1a00fcf80 0x7fe1a010df00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe190009fd0 tx=0x7fe19000b040 comp rx=0 tx=0).stop 2024-03-19T03:14:50.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.698+0000 7fe1a73a4700 1 -- 172.21.15.59:0/266939926 shutdown_connections 2024-03-19T03:14:50.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.698+0000 7fe1a73a4700 1 --2- 172.21.15.59:0/266939926 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fe18c0692a0 0x7fe18c06b750 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:50.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.698+0000 7fe1a73a4700 1 --2- 172.21.15.59:0/266939926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a00fbcd0 0x7fe1a010d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:50.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.698+0000 7fe1a73a4700 1 --2- 172.21.15.59:0/266939926 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1a00fcf80 0x7fe1a010df00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:50.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.698+0000 7fe1a73a4700 1 -- 172.21.15.59:0/266939926 >> 172.21.15.59:0/266939926 conn(0x7fe1a00f7430 msgr2=0x7fe1a0065b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:50.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.698+0000 7fe1a73a4700 1 -- 172.21.15.59:0/266939926 shutdown_connections 2024-03-19T03:14:50.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.698+0000 7fe1a73a4700 1 -- 172.21.15.59:0/266939926 wait complete. 2024-03-19T03:14:50.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.859+0000 7f90e56dd700 1 -- 172.21.15.59:0/981039492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90e00fdff0 msgr2=0x7f90e00fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:50.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.859+0000 7f90e56dd700 1 --2- 172.21.15.59:0/981039492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90e00fdff0 0x7f90e00fe460 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f90d4004940 tx=0x7f90d4009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:50.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.859+0000 7f90e56dd700 1 -- 172.21.15.59:0/981039492 shutdown_connections 2024-03-19T03:14:50.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.859+0000 7f90e56dd700 1 --2- 172.21.15.59:0/981039492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90e00fdff0 0x7f90e00fe460 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:50.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.859+0000 7f90e56dd700 1 --2- 172.21.15.59:0/981039492 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f90e00fce80 0x7f90e00fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:50.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.859+0000 7f90e56dd700 1 -- 172.21.15.59:0/981039492 >> 172.21.15.59:0/981039492 conn(0x7f90e00f8560 msgr2=0x7f90e00fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:50.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.860+0000 7f90e56dd700 1 -- 172.21.15.59:0/981039492 shutdown_connections 2024-03-19T03:14:50.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.860+0000 7f90e56dd700 1 -- 172.21.15.59:0/981039492 wait complete. 2024-03-19T03:14:50.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.860+0000 7f90e56dd700 1 Processor -- start 2024-03-19T03:14:50.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.860+0000 7f90e56dd700 1 -- start start 2024-03-19T03:14:50.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.860+0000 7f90e56dd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90e00fce80 0x7f90e0071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:50.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.860+0000 7f90e56dd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f90e00fdff0 0x7f90e0070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:50.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.860+0000 7f90e56dd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e00708c0 con 0x7f90e00fce80 2024-03-19T03:14:50.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.860+0000 7f90e56dd700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e0070a30 con 0x7f90e00fdff0 2024-03-19T03:14:50.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.861+0000 7f90deffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90e00fce80 0x7f90e0071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:50.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.861+0000 7f90de7fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f90e00fdff0 0x7f90e0070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:50.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.861+0000 7f90deffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90e00fce80 0x7f90e0071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60548/0 (socket says 172.21.15.59:60548) 2024-03-19T03:14:50.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.861+0000 7f90deffd700 1 -- 172.21.15.59:0/3469995429 learned_addr learned my addr 172.21.15.59:0/3469995429 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:14:50.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.861+0000 7f90deffd700 1 -- 172.21.15.59:0/3469995429 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f90e00fdff0 msgr2=0x7f90e0070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:50.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.861+0000 7f90deffd700 1 --2- 172.21.15.59:0/3469995429 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f90e00fdff0 0x7f90e0070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:50.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.861+0000 7f90deffd700 1 -- 172.21.15.59:0/3469995429 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d4005040 con 0x7f90e00fce80 2024-03-19T03:14:50.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.861+0000 7f90deffd700 1 --2- 172.21.15.59:0/3469995429 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90e00fce80 0x7f90e0071e40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f90d000eb40 tx=0x7f90d000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:50.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.861+0000 7f90c77fe700 1 -- 172.21.15.59:0/3469995429 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90d0007cc0 con 0x7f90e00fce80 2024-03-19T03:14:50.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.861+0000 7f90c77fe700 1 -- 172.21.15.59:0/3469995429 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90d000f040 con 0x7f90e00fce80 2024-03-19T03:14:50.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.861+0000 7f90c77fe700 1 -- 172.21.15.59:0/3469995429 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90d000ca10 con 0x7f90e00fce80 2024-03-19T03:14:50.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.861+0000 7f90e56dd700 1 -- 172.21.15.59:0/3469995429 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90e0070cb0 con 0x7f90e00fce80 2024-03-19T03:14:50.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.862+0000 7f90e56dd700 1 -- 172.21.15.59:0/3469995429 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90e01159e0 con 0x7f90e00fce80 2024-03-19T03:14:50.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.862+0000 7f90e56dd700 1 -- 172.21.15.59:0/3469995429 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90e005f5f0 con 0x7f90e00fce80 2024-03-19T03:14:50.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.863+0000 7f90c77fe700 1 -- 172.21.15.59:0/3469995429 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f90d000cc80 con 0x7f90e00fce80 2024-03-19T03:14:50.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.863+0000 7f90c77fe700 1 --2- 172.21.15.59:0/3469995429 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f90c80692a0 0x7f90c806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:14:50.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.864+0000 7f90c77fe700 1 -- 172.21.15.59:0/3469995429 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f90d0086d80 con 0x7f90e00fce80 2024-03-19T03:14:50.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.864+0000 7f90de7fc700 1 --2- 172.21.15.59:0/3469995429 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f90c80692a0 0x7f90c806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:14:50.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.864+0000 7f90de7fc700 1 --2- 172.21.15.59:0/3469995429 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f90c80692a0 0x7f90c806b750 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f90d4009b30 tx=0x7f90d4009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:14:50.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:49.868+0000 7f90c77fe700 1 -- 172.21.15.59:0/3469995429 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f90d004de50 con 0x7f90e00fce80 2024-03-19T03:14:50.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:50.165+0000 7f90e56dd700 1 -- 172.21.15.59:0/3469995429 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f90e0115c10 con 0x7f90e00fce80 2024-03-19T03:14:50.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:50.166+0000 7f90c77fe700 1 -- 172.21.15.59:0/3469995429 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f90d00530e0 con 0x7f90e00fce80 2024-03-19T03:14:50.167 INFO:teuthology.orchestra.run.smithi059.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-03-19T03:14:50.167 INFO:teuthology.orchestra.run.smithi059.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-03-19T03:14:50.167 INFO:teuthology.orchestra.run.smithi059.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-03-19T03:14:50.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:50.168+0000 7f90e56dd700 1 -- 172.21.15.59:0/3469995429 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f90c80692a0 msgr2=0x7f90c806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:50.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:50.168+0000 7f90e56dd700 1 --2- 172.21.15.59:0/3469995429 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f90c80692a0 0x7f90c806b750 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f90d4009b30 tx=0x7f90d4009e30 comp rx=0 tx=0).stop 2024-03-19T03:14:50.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:50.168+0000 7f90e56dd700 1 -- 172.21.15.59:0/3469995429 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90e00fce80 msgr2=0x7f90e0071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:14:50.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:50.168+0000 7f90e56dd700 1 --2- 172.21.15.59:0/3469995429 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90e00fce80 0x7f90e0071e40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f90d000eb40 tx=0x7f90d000b040 comp rx=0 tx=0).stop 2024-03-19T03:14:50.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:50.168+0000 7f90e56dd700 1 -- 172.21.15.59:0/3469995429 shutdown_connections 2024-03-19T03:14:50.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:50.168+0000 7f90e56dd700 1 --2- 172.21.15.59:0/3469995429 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f90c80692a0 0x7f90c806b750 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:50.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:50.168+0000 7f90e56dd700 1 --2- 172.21.15.59:0/3469995429 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90e00fce80 0x7f90e0071e40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:50.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:50.168+0000 7f90e56dd700 1 --2- 172.21.15.59:0/3469995429 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f90e00fdff0 0x7f90e0070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:14:50.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:50.168+0000 7f90e56dd700 1 -- 172.21.15.59:0/3469995429 >> 172.21.15.59:0/3469995429 conn(0x7f90e00f8560 msgr2=0x7f90e00f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:14:50.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:50.168+0000 7f90e56dd700 1 -- 172.21.15.59:0/3469995429 shutdown_connections 2024-03-19T03:14:50.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:14:50.168+0000 7f90e56dd700 1 -- 172.21.15.59:0/3469995429 wait complete. 2024-03-19T03:14:50.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:50 smithi059 ceph-mon[29133]: pgmap v222: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-19T03:14:50.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:50 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/1999324382' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-19T03:14:50.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:50 smithi176 ceph-mon[36450]: pgmap v222: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-19T03:14:50.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:50 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/1999324382' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-19T03:14:51.185 INFO:tasks.workunit.client.0.smithi059.stderr:Note: switching to 'e48b1f6c0608b2615e1890cc46286c3877d6e910'. 2024-03-19T03:14:51.185 INFO:tasks.workunit.client.0.smithi059.stderr: 2024-03-19T03:14:51.185 INFO:tasks.workunit.client.0.smithi059.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-19T03:14:51.185 INFO:tasks.workunit.client.0.smithi059.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-19T03:14:51.185 INFO:tasks.workunit.client.0.smithi059.stderr:state without impacting any branches by switching back to a branch. 2024-03-19T03:14:51.185 INFO:tasks.workunit.client.0.smithi059.stderr: 2024-03-19T03:14:51.185 INFO:tasks.workunit.client.0.smithi059.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-19T03:14:51.186 INFO:tasks.workunit.client.0.smithi059.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-19T03:14:51.186 INFO:tasks.workunit.client.0.smithi059.stderr: 2024-03-19T03:14:51.186 INFO:tasks.workunit.client.0.smithi059.stderr: git switch -c 2024-03-19T03:14:51.186 INFO:tasks.workunit.client.0.smithi059.stderr: 2024-03-19T03:14:51.186 INFO:tasks.workunit.client.0.smithi059.stderr:Or undo this operation with: 2024-03-19T03:14:51.186 INFO:tasks.workunit.client.0.smithi059.stderr: 2024-03-19T03:14:51.186 INFO:tasks.workunit.client.0.smithi059.stderr: git switch - 2024-03-19T03:14:51.186 INFO:tasks.workunit.client.0.smithi059.stderr: 2024-03-19T03:14:51.186 INFO:tasks.workunit.client.0.smithi059.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-19T03:14:51.186 INFO:tasks.workunit.client.0.smithi059.stderr: 2024-03-19T03:14:51.186 INFO:tasks.workunit.client.0.smithi059.stderr:HEAD is now at e48b1f6c060 testfix 2024-03-19T03:14:51.190 DEBUG:teuthology.orchestra.run.smithi059:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-19T03:14:51.213 INFO:tasks.workunit.client.0.smithi059.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-19T03:14:51.215 INFO:tasks.workunit.client.0.smithi059.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-19T03:14:51.215 INFO:tasks.workunit.client.0.smithi059.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-19T03:14:51.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:51 smithi059 ceph-mon[29133]: from='client.24389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:14:51.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:51 smithi059 ceph-mon[29133]: from='client.? 172.21.15.59:0/3469995429' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-19T03:14:51.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:51 smithi176 ceph-mon[36450]: from='client.24389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:14:51.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:51 smithi176 ceph-mon[36450]: from='client.? 172.21.15.59:0/3469995429' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-19T03:14:51.959 INFO:tasks.workunit.client.0.smithi059.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-19T03:14:52.038 INFO:tasks.workunit.client.0.smithi059.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-19T03:14:52.083 INFO:tasks.workunit.client.0.smithi059.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-19T03:14:52.084 INFO:tasks.workunit.client.0.smithi059.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-19T03:14:52.085 INFO:tasks.workunit.client.0.smithi059.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-19T03:14:52.203 INFO:tasks.workunit.client.0.smithi059.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-19T03:14:52.207 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-19T03:14:52.207 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-19T03:14:52.262 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-03-19T03:14:52.264 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-19T03:14:52.264 DEBUG:teuthology.orchestra.run.smithi059:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e48b1f6c0608b2615e1890cc46286c3877d6e910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-03-19T03:14:52.323 INFO:tasks.workunit.client.0.smithi059.stderr:+ mkdir -p fsstress 2024-03-19T03:14:52.325 INFO:tasks.workunit.client.0.smithi059.stderr:+ pushd fsstress 2024-03-19T03:14:52.325 INFO:tasks.workunit.client.0.smithi059.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-19T03:14:52.326 INFO:tasks.workunit.client.0.smithi059.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-19T03:14:52.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:52 smithi059 ceph-mon[29133]: pgmap v223: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-19T03:14:52.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:52 smithi176 ceph-mon[36450]: pgmap v223: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-19T03:14:53.016 INFO:tasks.workunit.client.0.smithi059.stderr:+ tar xzf ltp-full.tgz 2024-03-19T03:14:54.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:54 smithi059 ceph-mon[29133]: pgmap v224: 65 pgs: 65 active+clean; 5.9 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 453 KiB/s wr, 0 op/s 2024-03-19T03:14:54.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:54 smithi176 ceph-mon[36450]: pgmap v224: 65 pgs: 65 active+clean; 5.9 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 453 KiB/s wr, 0 op/s 2024-03-19T03:14:56.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:56 smithi059 ceph-mon[29133]: pgmap v225: 65 pgs: 65 active+clean; 16 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s wr, 107 op/s 2024-03-19T03:14:56.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:56 smithi176 ceph-mon[36450]: pgmap v225: 65 pgs: 65 active+clean; 16 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s wr, 107 op/s 2024-03-19T03:14:58.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:14:58 smithi059 ceph-mon[29133]: pgmap v226: 65 pgs: 65 active+clean; 30 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.6 MiB/s wr, 178 op/s 2024-03-19T03:14:58.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:14:58 smithi176 ceph-mon[36450]: pgmap v226: 65 pgs: 65 active+clean; 30 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.6 MiB/s wr, 178 op/s 2024-03-19T03:15:00.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:15:00 smithi059 ceph-mon[29133]: pgmap v227: 65 pgs: 65 active+clean; 62 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 5.4 MiB/s wr, 491 op/s 2024-03-19T03:15:00.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:15:00 smithi176 ceph-mon[36450]: pgmap v227: 65 pgs: 65 active+clean; 62 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 5.4 MiB/s wr, 491 op/s 2024-03-19T03:15:01.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:15:01 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:15:01.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:15:01 smithi059 ceph-mon[29133]: pgmap v228: 65 pgs: 65 active+clean; 67 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 5.8 MiB/s wr, 525 op/s 2024-03-19T03:15:01.877 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:15:01 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-19T03:15:01.877 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:15:01 smithi176 ceph-mon[36450]: pgmap v228: 65 pgs: 65 active+clean; 67 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 5.8 MiB/s wr, 525 op/s 2024-03-19T03:15:04.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:15:04 smithi059 ceph-mon[29133]: pgmap v229: 65 pgs: 65 active+clean; 87 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 7.6 MiB/s wr, 585 op/s 2024-03-19T03:15:04.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:15:04 smithi176 ceph-mon[36450]: pgmap v229: 65 pgs: 65 active+clean; 87 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 7.6 MiB/s wr, 585 op/s 2024-03-19T03:15:06.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:15:06 smithi059 ceph-mon[29133]: pgmap v230: 65 pgs: 65 active+clean; 87 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 7.1 MiB/s wr, 585 op/s 2024-03-19T03:15:06.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:15:06 smithi176 ceph-mon[36450]: pgmap v230: 65 pgs: 65 active+clean; 87 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 7.1 MiB/s wr, 585 op/s 2024-03-19T03:15:08.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:15:08 smithi059 ceph-mon[29133]: pgmap v231: 65 pgs: 65 active+clean; 89 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 6.4 MiB/s wr, 512 op/s 2024-03-19T03:15:08.377 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:15:08 smithi176 ceph-mon[36450]: pgmap v231: 65 pgs: 65 active+clean; 89 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 6.4 MiB/s wr, 512 op/s 2024-03-19T03:15:09.607 INFO:tasks.workunit.client.1.smithi176.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-03-19T03:15:09.607 INFO:tasks.workunit.client.1.smithi176.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-19T03:15:09.607 INFO:tasks.workunit.client.1.smithi176.stderr:+ make 2024-03-19T03:15:10.148 INFO:tasks.workunit.client.1.smithi176.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-03-19T03:15:10.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:15:10 smithi059 ceph-mon[29133]: pgmap v232: 65 pgs: 65 active+clean; 115 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 7.4 MiB/s wr, 614 op/s 2024-03-19T03:15:10.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:15:10 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:15:10.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:15:10 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi176.ycmqbw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T03:15:10.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:15:10 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T03:15:10.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:15:10 smithi059 ceph-mon[29133]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:15:10.373 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:15:10 smithi176 ceph-mon[36450]: pgmap v232: 65 pgs: 65 active+clean; 115 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 7.4 MiB/s wr, 614 op/s 2024-03-19T03:15:10.373 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:15:10 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' 2024-03-19T03:15:10.373 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:15:10 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi176.ycmqbw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T03:15:10.373 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:15:10 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T03:15:10.374 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Mar 19 03:15:10 smithi176 ceph-mon[36450]: from='mgr.14214 172.21.15.59:0/123058768' entity='mgr.smithi059.azpecc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:15:20.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.288+0000 7f2e1980c700 1 -- 172.21.15.59:0/4233178797 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e140662e0 msgr2=0x7f2e140ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:15:20.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.288+0000 7f2e1980c700 1 --2- 172.21.15.59:0/4233178797 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e140662e0 0x7f2e140ff9c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2e08004940 tx=0x7f2e08009e30 comp rx=0 tx=0).stop 2024-03-19T03:15:20.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.288+0000 7f2e1980c700 1 -- 172.21.15.59:0/4233178797 shutdown_connections 2024-03-19T03:15:20.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.288+0000 7f2e1980c700 1 --2- 172.21.15.59:0/4233178797 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e140fff00 0x7f2e14102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:15:20.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.288+0000 7f2e1980c700 1 --2- 172.21.15.59:0/4233178797 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e140662e0 0x7f2e140ff9c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:15:20.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.288+0000 7f2e1980c700 1 -- 172.21.15.59:0/4233178797 >> 172.21.15.59:0/4233178797 conn(0x7f2e140f7430 msgr2=0x7f2e140f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:15:20.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.288+0000 7f2e1980c700 1 -- 172.21.15.59:0/4233178797 shutdown_connections 2024-03-19T03:15:20.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.288+0000 7f2e1980c700 1 -- 172.21.15.59:0/4233178797 wait complete. 2024-03-19T03:15:20.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.289+0000 7f2e1980c700 1 Processor -- start 2024-03-19T03:15:20.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.289+0000 7f2e1980c700 1 -- start start 2024-03-19T03:15:20.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.289+0000 7f2e1980c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e140662e0 0x7f2e140fd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:15:20.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.289+0000 7f2e1980c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e140fff00 0x7f2e140fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:15:20.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.289+0000 7f2e1980c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e140fbf00 con 0x7f2e140fff00 2024-03-19T03:15:20.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.289+0000 7f2e1980c700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e140fc070 con 0x7f2e140662e0 2024-03-19T03:15:20.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.289+0000 7f2e127fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e140fff00 0x7f2e140fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:15:20.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.289+0000 7f2e127fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e140fff00 0x7f2e140fb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60514/0 (socket says 172.21.15.59:60514) 2024-03-19T03:15:20.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.289+0000 7f2e127fc700 1 -- 172.21.15.59:0/3708023512 learned_addr learned my addr 172.21.15.59:0/3708023512 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:15:20.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.290+0000 7f2e127fc700 1 -- 172.21.15.59:0/3708023512 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e140662e0 msgr2=0x7f2e140fd3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-19T03:15:20.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.290+0000 7f2e127fc700 1 --2- 172.21.15.59:0/3708023512 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e140662e0 0x7f2e140fd3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:15:20.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.290+0000 7f2e127fc700 1 -- 172.21.15.59:0/3708023512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e08005040 con 0x7f2e140fff00 2024-03-19T03:15:20.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.290+0000 7f2e127fc700 1 --2- 172.21.15.59:0/3708023512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e140fff00 0x7f2e140fb8c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2e0400b580 tx=0x7f2e040091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:15:20.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.290+0000 7f2dfb7fe700 1 -- 172.21.15.59:0/3708023512 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e0400f6f0 con 0x7f2e140fff00 2024-03-19T03:15:20.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.290+0000 7f2dfb7fe700 1 -- 172.21.15.59:0/3708023512 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e0400fd20 con 0x7f2e140fff00 2024-03-19T03:15:20.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.290+0000 7f2dfb7fe700 1 -- 172.21.15.59:0/3708023512 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e04011ea0 con 0x7f2e140fff00 2024-03-19T03:15:20.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.290+0000 7f2e1980c700 1 -- 172.21.15.59:0/3708023512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e140fc350 con 0x7f2e140fff00 2024-03-19T03:15:20.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.290+0000 7f2e1980c700 1 -- 172.21.15.59:0/3708023512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e14115b80 con 0x7f2e140fff00 2024-03-19T03:15:20.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.291+0000 7f2e1980c700 1 -- 172.21.15.59:0/3708023512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e1405f5f0 con 0x7f2e140fff00 2024-03-19T03:15:20.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.291+0000 7f2dfb7fe700 1 -- 172.21.15.59:0/3708023512 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f2e0400f850 con 0x7f2e140fff00 2024-03-19T03:15:20.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.292+0000 7f2dfb7fe700 1 --2- 172.21.15.59:0/3708023512 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2dfc0692a0 0x7f2dfc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:15:20.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.292+0000 7f2dfb7fe700 1 -- 172.21.15.59:0/3708023512 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f2e04086d80 con 0x7f2e140fff00 2024-03-19T03:15:20.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.292+0000 7f2e12ffd700 1 --2- 172.21.15.59:0/3708023512 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2dfc0692a0 0x7f2dfc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:15:20.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.292+0000 7f2e12ffd700 1 --2- 172.21.15.59:0/3708023512 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2dfc0692a0 0x7f2dfc06b750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2e0800b3b0 tx=0x7f2e08009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:15:20.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.296+0000 7f2dfb7fe700 1 -- 172.21.15.59:0/3708023512 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f2e04053150 con 0x7f2e140fff00 2024-03-19T03:15:20.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.530+0000 7f2e1980c700 1 -- 172.21.15.59:0/3708023512 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e140f8f70 con 0x7f2dfc0692a0 2024-03-19T03:15:20.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.531+0000 7f2dfb7fe700 1 -- 172.21.15.59:0/3708023512 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+349 (secure 0 0 0) 0x7f2e140f8f70 con 0x7f2dfc0692a0 2024-03-19T03:15:20.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.533+0000 7f2e1980c700 1 -- 172.21.15.59:0/3708023512 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2dfc0692a0 msgr2=0x7f2dfc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:15:20.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.533+0000 7f2e1980c700 1 --2- 172.21.15.59:0/3708023512 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2dfc0692a0 0x7f2dfc06b750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2e0800b3b0 tx=0x7f2e08009e30 comp rx=0 tx=0).stop 2024-03-19T03:15:20.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.533+0000 7f2e1980c700 1 -- 172.21.15.59:0/3708023512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e140fff00 msgr2=0x7f2e140fb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:15:20.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.533+0000 7f2e1980c700 1 --2- 172.21.15.59:0/3708023512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e140fff00 0x7f2e140fb8c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2e0400b580 tx=0x7f2e040091a0 comp rx=0 tx=0).stop 2024-03-19T03:15:20.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.533+0000 7f2e1980c700 1 -- 172.21.15.59:0/3708023512 shutdown_connections 2024-03-19T03:15:20.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.533+0000 7f2e1980c700 1 --2- 172.21.15.59:0/3708023512 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f2dfc0692a0 0x7f2dfc06b750 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:15:20.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.533+0000 7f2e1980c700 1 --2- 172.21.15.59:0/3708023512 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e140662e0 0x7f2e140fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:15:20.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.533+0000 7f2e1980c700 1 --2- 172.21.15.59:0/3708023512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e140fff00 0x7f2e140fb8c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:15:20.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.533+0000 7f2e1980c700 1 -- 172.21.15.59:0/3708023512 >> 172.21.15.59:0/3708023512 conn(0x7f2e140f7430 msgr2=0x7f2e140f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:15:20.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.533+0000 7f2e1980c700 1 -- 172.21.15.59:0/3708023512 shutdown_connections 2024-03-19T03:15:20.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.533+0000 7f2e1980c700 1 -- 172.21.15.59:0/3708023512 wait complete. 2024-03-19T03:15:20.544 INFO:teuthology.orchestra.run.smithi059.stdout:true 2024-03-19T03:15:20.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.636+0000 7f525d898700 1 -- 172.21.15.59:0/424867275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5258066f10 msgr2=0x7f52580ffa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:15:20.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.636+0000 7f525d898700 1 --2- 172.21.15.59:0/424867275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5258066f10 0x7f52580ffa20 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5248004940 tx=0x7f5248009e30 comp rx=0 tx=0).stop 2024-03-19T03:15:20.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.636+0000 7f525d898700 1 -- 172.21.15.59:0/424867275 shutdown_connections 2024-03-19T03:15:20.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.636+0000 7f525d898700 1 --2- 172.21.15.59:0/424867275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52580fff60 0x7f5258102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:15:20.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.636+0000 7f525d898700 1 --2- 172.21.15.59:0/424867275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5258066f10 0x7f52580ffa20 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:15:20.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.636+0000 7f525d898700 1 -- 172.21.15.59:0/424867275 >> 172.21.15.59:0/424867275 conn(0x7f52580f7430 msgr2=0x7f52580f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:15:20.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.637+0000 7f525d898700 1 -- 172.21.15.59:0/424867275 shutdown_connections 2024-03-19T03:15:20.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.637+0000 7f525d898700 1 -- 172.21.15.59:0/424867275 wait complete. 2024-03-19T03:15:20.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.637+0000 7f525d898700 1 Processor -- start 2024-03-19T03:15:20.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.637+0000 7f525d898700 1 -- start start 2024-03-19T03:15:20.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.637+0000 7f525d898700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5258066f10 0x7f525810da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:15:20.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.638+0000 7f525d898700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52580fff60 0x7f525810df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:15:20.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.638+0000 7f525d898700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525810e5e0 con 0x7f52580fff60 2024-03-19T03:15:20.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.638+0000 7f525d898700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5258112f80 con 0x7f5258066f10 2024-03-19T03:15:20.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.638+0000 7f52567fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52580fff60 0x7f525810df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:15:20.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.638+0000 7f52567fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52580fff60 0x7f525810df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60522/0 (socket says 172.21.15.59:60522) 2024-03-19T03:15:20.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.638+0000 7f52567fc700 1 -- 172.21.15.59:0/3662082700 learned_addr learned my addr 172.21.15.59:0/3662082700 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:15:20.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.638+0000 7f52567fc700 1 -- 172.21.15.59:0/3662082700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5258066f10 msgr2=0x7f525810da10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-19T03:15:20.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.638+0000 7f52567fc700 1 --2- 172.21.15.59:0/3662082700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5258066f10 0x7f525810da10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:15:20.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.638+0000 7f52567fc700 1 -- 172.21.15.59:0/3662082700 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5248005040 con 0x7f52580fff60 2024-03-19T03:15:20.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.638+0000 7f52567fc700 1 --2- 172.21.15.59:0/3662082700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52580fff60 0x7f525810df50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f524c00b580 tx=0x7f524c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:15:20.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.638+0000 7f523f7fe700 1 -- 172.21.15.59:0/3662082700 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f524c00f6f0 con 0x7f52580fff60 2024-03-19T03:15:20.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.638+0000 7f523f7fe700 1 -- 172.21.15.59:0/3662082700 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f524c00fd20 con 0x7f52580fff60 2024-03-19T03:15:20.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.638+0000 7f525d898700 1 -- 172.21.15.59:0/3662082700 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52581131e0 con 0x7f52580fff60 2024-03-19T03:15:20.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.638+0000 7f523f7fe700 1 -- 172.21.15.59:0/3662082700 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f524c011ea0 con 0x7f52580fff60 2024-03-19T03:15:20.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.638+0000 7f525d898700 1 -- 172.21.15.59:0/3662082700 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5258113750 con 0x7f52580fff60 2024-03-19T03:15:20.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.639+0000 7f525d898700 1 -- 172.21.15.59:0/3662082700 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f525804c710 con 0x7f52580fff60 2024-03-19T03:15:20.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.640+0000 7f523f7fe700 1 -- 172.21.15.59:0/3662082700 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f524c00f850 con 0x7f52580fff60 2024-03-19T03:15:20.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.640+0000 7f523f7fe700 1 --2- 172.21.15.59:0/3662082700 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f52400692a0 0x7f524006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:15:20.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.640+0000 7f523f7fe700 1 -- 172.21.15.59:0/3662082700 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f524c086e50 con 0x7f52580fff60 2024-03-19T03:15:20.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.640+0000 7f5256ffd700 1 --2- 172.21.15.59:0/3662082700 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f52400692a0 0x7f524006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:15:20.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.641+0000 7f5256ffd700 1 --2- 172.21.15.59:0/3662082700 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f52400692a0 0x7f524006b750 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f524800b970 tx=0x7f5248009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:15:20.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.645+0000 7f523f7fe700 1 -- 172.21.15.59:0/3662082700 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7f524c056c10 con 0x7f52580fff60 2024-03-19T03:15:20.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.866+0000 7f525d898700 1 -- 172.21.15.59:0/3662082700 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f52580f8f70 con 0x7f52400692a0 2024-03-19T03:15:20.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.867+0000 7f523f7fe700 1 -- 172.21.15.59:0/3662082700 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+349 (secure 0 0 0) 0x7f52580f8f70 con 0x7f52400692a0 2024-03-19T03:15:20.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.869+0000 7f525d898700 1 -- 172.21.15.59:0/3662082700 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f52400692a0 msgr2=0x7f524006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:15:20.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.869+0000 7f525d898700 1 --2- 172.21.15.59:0/3662082700 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f52400692a0 0x7f524006b750 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f524800b970 tx=0x7f5248009e30 comp rx=0 tx=0).stop 2024-03-19T03:15:20.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.869+0000 7f525d898700 1 -- 172.21.15.59:0/3662082700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52580fff60 msgr2=0x7f525810df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:15:20.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.869+0000 7f525d898700 1 --2- 172.21.15.59:0/3662082700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52580fff60 0x7f525810df50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f524c00b580 tx=0x7f524c0091a0 comp rx=0 tx=0).stop 2024-03-19T03:15:20.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.869+0000 7f525d898700 1 -- 172.21.15.59:0/3662082700 shutdown_connections 2024-03-19T03:15:20.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.869+0000 7f525d898700 1 --2- 172.21.15.59:0/3662082700 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7f52400692a0 0x7f524006b750 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:15:20.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.869+0000 7f525d898700 1 --2- 172.21.15.59:0/3662082700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5258066f10 0x7f525810da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:15:20.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.869+0000 7f525d898700 1 --2- 172.21.15.59:0/3662082700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52580fff60 0x7f525810df50 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:15:20.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.869+0000 7f525d898700 1 -- 172.21.15.59:0/3662082700 >> 172.21.15.59:0/3662082700 conn(0x7f52580f7430 msgr2=0x7f52580f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:15:20.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.869+0000 7f525d898700 1 -- 172.21.15.59:0/3662082700 shutdown_connections 2024-03-19T03:15:20.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.869+0000 7f525d898700 1 -- 172.21.15.59:0/3662082700 wait complete. 2024-03-19T03:15:20.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.974+0000 7fc6efa10700 1 -- 172.21.15.59:0/2138715473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6e80fed90 msgr2=0x7fc6e80faed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:15:20.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.974+0000 7fc6efa10700 1 --2- 172.21.15.59:0/2138715473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6e80fed90 0x7fc6e80faed0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc004940 tx=0x7fc6dc009e30 comp rx=0 tx=0).stop 2024-03-19T03:15:20.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.974+0000 7fc6efa10700 1 -- 172.21.15.59:0/2138715473 shutdown_connections 2024-03-19T03:15:20.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.974+0000 7fc6efa10700 1 --2- 172.21.15.59:0/2138715473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6e80fed90 0x7fc6e80faed0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:15:20.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.974+0000 7fc6efa10700 1 --2- 172.21.15.59:0/2138715473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6e80fe440 0x7fc6e80fe850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:15:20.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.974+0000 7fc6efa10700 1 -- 172.21.15.59:0/2138715473 >> 172.21.15.59:0/2138715473 conn(0x7fc6e80f6bc0 msgr2=0x7fc6e80f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:15:20.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.974+0000 7fc6efa10700 1 -- 172.21.15.59:0/2138715473 shutdown_connections 2024-03-19T03:15:20.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.974+0000 7fc6efa10700 1 -- 172.21.15.59:0/2138715473 wait complete. 2024-03-19T03:15:20.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.974+0000 7fc6efa10700 1 Processor -- start 2024-03-19T03:15:20.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.974+0000 7fc6efa10700 1 -- start start 2024-03-19T03:15:20.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.975+0000 7fc6efa10700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6e80fe440 0x7fc6e80fda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:15:20.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.975+0000 7fc6efa10700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6e80fed90 0x7fc6e80fc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:15:20.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.975+0000 7fc6efa10700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e80fc640 con 0x7fc6e80fed90 2024-03-19T03:15:20.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.975+0000 7fc6efa10700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e80fc7b0 con 0x7fc6e80fe440 2024-03-19T03:15:20.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.975+0000 7fc6ecfab700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6e80fed90 0x7fc6e80fc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:15:20.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.975+0000 7fc6ecfab700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6e80fed90 0x7fc6e80fc100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60542/0 (socket says 172.21.15.59:60542) 2024-03-19T03:15:20.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.975+0000 7fc6ecfab700 1 -- 172.21.15.59:0/2511186269 learned_addr learned my addr 172.21.15.59:0/2511186269 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-19T03:15:20.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.975+0000 7fc6ecfab700 1 -- 172.21.15.59:0/2511186269 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6e80fe440 msgr2=0x7fc6e80fda50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-19T03:15:20.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.975+0000 7fc6ecfab700 1 --2- 172.21.15.59:0/2511186269 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6e80fe440 0x7fc6e80fda50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:15:20.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.975+0000 7fc6ecfab700 1 -- 172.21.15.59:0/2511186269 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6dc005040 con 0x7fc6e80fed90 2024-03-19T03:15:20.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.975+0000 7fc6ecfab700 1 --2- 172.21.15.59:0/2511186269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6e80fed90 0x7fc6e80fc100 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc009040 tx=0x7fc6dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:15:20.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.975+0000 7fc6e5ffb700 1 -- 172.21.15.59:0/2511186269 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6dc00e070 con 0x7fc6e80fed90 2024-03-19T03:15:20.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.976+0000 7fc6e5ffb700 1 -- 172.21.15.59:0/2511186269 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc6dc009430 con 0x7fc6e80fed90 2024-03-19T03:15:20.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.976+0000 7fc6efa10700 1 -- 172.21.15.59:0/2511186269 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e80fca30 con 0x7fc6e80fed90 2024-03-19T03:15:20.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.976+0000 7fc6e5ffb700 1 -- 172.21.15.59:0/2511186269 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6dc01c400 con 0x7fc6e80fed90 2024-03-19T03:15:20.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.976+0000 7fc6efa10700 1 -- 172.21.15.59:0/2511186269 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e8115ac0 con 0x7fc6e80fed90 2024-03-19T03:15:20.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.977+0000 7fc6efa10700 1 -- 172.21.15.59:0/2511186269 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6e804c710 con 0x7fc6e80fed90 2024-03-19T03:15:20.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.977+0000 7fc6e5ffb700 1 -- 172.21.15.59:0/2511186269 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc6dc013420 con 0x7fc6e80fed90 2024-03-19T03:15:20.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.977+0000 7fc6e5ffb700 1 --2- 172.21.15.59:0/2511186269 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc6d00692a0 0x7fc6d006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:15:20.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.978+0000 7fc6e5ffb700 1 -- 172.21.15.59:0/2511186269 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fc6dc0889a0 con 0x7fc6e80fed90 2024-03-19T03:15:20.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.978+0000 7fc6ed7ac700 1 --2- 172.21.15.59:0/2511186269 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc6d00692a0 0x7fc6d006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:15:20.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.978+0000 7fc6ed7ac700 1 --2- 172.21.15.59:0/2511186269 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc6d00692a0 0x7fc6d006b750 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0009740 tx=0x7fc6e0006c60 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:15:20.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:20.983+0000 7fc6e5ffb700 1 -- 172.21.15.59:0/2511186269 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175674 (secure 0 0 0) 0x7fc6dc058760 con 0x7fc6e80fed90 2024-03-19T03:15:21.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:21.183+0000 7fc6efa10700 1 -- 172.21.15.59:0/2511186269 --> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6e8102490 con 0x7fc6d00692a0 2024-03-19T03:15:21.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:21.193+0000 7fc6e5ffb700 1 -- 172.21.15.59:0/2511186269 <== mgr.14214 v2:172.21.15.59:6800/2563034496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7fc6e8102490 con 0x7fc6d00692a0 2024-03-19T03:15:21.194 INFO:teuthology.orchestra.run.smithi059.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-19T03:15:21.195 INFO:teuthology.orchestra.run.smithi059.stdout:alertmanager.smithi059 smithi059 *:9093,9094 running (6m) 96s ago 7m 27.3M - 0.25.0 c8568f914cd2 40a81ac05a90 2024-03-19T03:15:21.195 INFO:teuthology.orchestra.run.smithi059.stdout:crash.smithi059 smithi059 running (7m) 96s ago 7m 6669k - 17.2.7-700-gb4843bd7 810955603526 3090e46c4bff 2024-03-19T03:15:21.195 INFO:teuthology.orchestra.run.smithi059.stdout:crash.smithi176 smithi176 running (6m) 102s ago 6m 6660k - 17.2.7-700-gb4843bd7 810955603526 2df9a0e61e4c 2024-03-19T03:15:21.195 INFO:teuthology.orchestra.run.smithi059.stdout:grafana.smithi059 smithi059 *:3000 running (5m) 96s ago 7m 85.4M - 9.4.7 954c08fa6188 b3394e3d4026 2024-03-19T03:15:21.195 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi059.doeuxg smithi059 running (114s) 96s ago 114s 14.5M - 17.2.7-700-gb4843bd7 810955603526 18d717af1972 2024-03-19T03:15:21.195 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi059.hmylwa smithi059 running (2m) 96s ago 2m 14.6M - 17.2.7-700-gb4843bd7 810955603526 d4acd92451b4 2024-03-19T03:15:21.195 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi176.bochhq smithi176 running (118s) 102s ago 117s 14.3M - 17.2.7-700-gb4843bd7 810955603526 05e9bc441018 2024-03-19T03:15:21.195 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi176.gisnqc smithi176 running (112s) 102s ago 111s 12.2M - 17.2.7-700-gb4843bd7 810955603526 75982373b447 2024-03-19T03:15:21.195 INFO:teuthology.orchestra.run.smithi059.stdout:mgr.smithi059.azpecc smithi059 *:9283 running (8m) 96s ago 8m 480M - 17.2.7-700-gb4843bd7 810955603526 441ca0cfcfac 2024-03-19T03:15:21.195 INFO:teuthology.orchestra.run.smithi059.stdout:mgr.smithi176.ycmqbw smithi176 *:8443,9283 running (6m) 102s ago 6m 433M - 17.2.7-700-gb4843bd7 810955603526 ac8bc9c6ec9b 2024-03-19T03:15:21.195 INFO:teuthology.orchestra.run.smithi059.stdout:mon.smithi059 smithi059 running (9m) 96s ago 9m 49.9M 2048M 17.2.7-700-gb4843bd7 810955603526 c8bc018eaa37 2024-03-19T03:15:21.195 INFO:teuthology.orchestra.run.smithi059.stdout:mon.smithi176 smithi176 running (6m) 102s ago 6m 33.4M 2048M 17.2.7-700-gb4843bd7 810955603526 4130cce6243b 2024-03-19T03:15:21.195 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter.smithi059 smithi059 *:9100 running (7m) 96s ago 7m 16.1M - 1.5.0 0da6a335fe13 d3fca5e9a667 2024-03-19T03:15:21.195 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter.smithi176 smithi176 *:9100 running (6m) 102s ago 6m 14.7M - 1.5.0 0da6a335fe13 35389a3c33fd 2024-03-19T03:15:21.196 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 smithi059 running (5m) 96s ago 5m 46.3M 4289M 17.2.7-700-gb4843bd7 810955603526 2511279e8852 2024-03-19T03:15:21.196 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 smithi059 running (5m) 96s ago 5m 45.0M 4289M 17.2.7-700-gb4843bd7 810955603526 cd85f348d650 2024-03-19T03:15:21.196 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 smithi059 running (4m) 96s ago 4m 47.0M 4289M 17.2.7-700-gb4843bd7 810955603526 363582327fde 2024-03-19T03:15:21.196 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 smithi176 running (4m) 102s ago 4m 43.5M 5312M 17.2.7-700-gb4843bd7 810955603526 436c587b5265 2024-03-19T03:15:21.196 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 smithi176 running (3m) 102s ago 3m 43.1M 5312M 17.2.7-700-gb4843bd7 810955603526 80d2dd2cd18b 2024-03-19T03:15:21.196 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 smithi176 running (3m) 102s ago 3m 41.6M 5312M 17.2.7-700-gb4843bd7 810955603526 dd0f98b5ec05 2024-03-19T03:15:21.196 INFO:teuthology.orchestra.run.smithi059.stdout:prometheus.smithi059 smithi059 *:9095 running (5m) 96s ago 6m 54.1M - 2.43.0 a07b618ecd1d 1782702019d3 2024-03-19T03:15:21.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:21.195+0000 7fc6efa10700 1 -- 172.21.15.59:0/2511186269 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc6d00692a0 msgr2=0x7fc6d006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:15:21.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:21.195+0000 7fc6efa10700 1 --2- 172.21.15.59:0/2511186269 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc6d00692a0 0x7fc6d006b750 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0009740 tx=0x7fc6e0006c60 comp rx=0 tx=0).stop 2024-03-19T03:15:21.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:21.195+0000 7fc6efa10700 1 -- 172.21.15.59:0/2511186269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6e80fed90 msgr2=0x7fc6e80fc100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:15:21.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:21.195+0000 7fc6efa10700 1 --2- 172.21.15.59:0/2511186269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6e80fed90 0x7fc6e80fc100 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc009040 tx=0x7fc6dc009e30 comp rx=0 tx=0).stop 2024-03-19T03:15:21.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:21.195+0000 7fc6efa10700 1 -- 172.21.15.59:0/2511186269 shutdown_connections 2024-03-19T03:15:21.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:21.195+0000 7fc6efa10700 1 --2- 172.21.15.59:0/2511186269 >> [v2:172.21.15.59:6800/2563034496,v1:172.21.15.59:6801/2563034496] conn(0x7fc6d00692a0 0x7fc6d006b750 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:15:21.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:21.195+0000 7fc6efa10700 1 --2- 172.21.15.59:0/2511186269 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6e80fe440 0x7fc6e80fda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:15:21.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:21.195+0000 7fc6efa10700 1 --2- 172.21.15.59:0/2511186269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6e80fed90 0x7fc6e80fc100 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:15:21.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:21.195+0000 7fc6efa10700 1 -- 172.21.15.59:0/2511186269 >> 172.21.15.59:0/2511186269 conn(0x7fc6e80f6bc0 msgr2=0x7fc6e80f8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:15:21.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:21.195+0000 7fc6efa10700 1 -- 172.21.15.59:0/2511186269 shutdown_connections 2024-03-19T03:15:21.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:15:21.195+0000 7fc6efa10700 1 -- 172.21.15.59:0/2511186269 wait complete. 2024-03-19T03:15:31.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:31.229+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:31.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:31.229+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:31.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:31.229+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:32.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:32.255+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:32.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:32.255+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:32.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:32.255+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:33.508 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:33.215+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:33.508 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:33.215+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:33.508 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:33.215+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:33.508 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:33.283+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:33.508 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:33.283+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:33.509 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:33.283+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:33.509 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:33.229+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:33.509 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:33.229+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:33.509 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:33.229+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:34.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:34.257+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:34.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:34.257+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:34.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:34.257+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:34.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:34.205+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:34.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:34.205+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:34.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:34.205+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:34.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:34.242+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:34.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:34.242+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:34.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:34.242+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:35.491 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:35.282+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:35.491 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:35.282+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:35.491 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:35.282+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:35.491 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:35.297+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:35.492 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:35.297+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:35.492 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:35.297+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:35.492 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:35.232+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:35.492 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:35.232+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:35.492 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:35.232+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:36.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:36.331+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:36.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:36.331+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:36.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:36.331+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:36.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:36.280+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:36.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:36.280+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:36.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:36.280+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:36.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:36.190+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:36.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:36.190+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:36.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:36.190+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:37.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:37.319+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:37.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:37.319+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:37.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:37.319+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:37.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:37.293+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:37.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:37.293+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:37.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:37.293+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:37.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:37.199+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:37.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:37.199+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:37.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:37.199+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:38.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:38.319+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:38.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:38.319+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:38.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:38.319+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:38.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:38.294+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:38.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:38.294+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:38.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:38.294+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:38.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:38.194+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:38.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:38.194+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:38.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:38.194+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:39.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:39.145+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:39.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:39.145+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:39.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:39.145+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:39.314+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:39.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:39.314+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:39.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:39.314+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:39.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:39.333+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:39.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:39.333+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:39.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:39.333+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:40.357 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:40.274+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:40.357 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:40.274+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:40.358 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:40.274+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:40.358 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:40.105+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:40.358 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:40.105+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:40.358 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:40.105+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:40.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:40.356+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:40.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:40.356+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:40.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:40.356+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:41.386 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:41.256+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:41.387 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:41.256+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:41.387 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:41.256+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:41.387 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:41.125+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:41.387 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:41.125+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:41.387 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:41.125+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:41.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:41.385+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:41.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:41.385+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:41.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:41.385+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:42.363 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:42.249+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:42.363 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:42.250+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:42.363 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:42.250+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:42.363 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:42.096+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:42.363 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:42.096+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:42.363 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:42.096+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:42.361+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:42.361+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:42.361+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:43.383 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:43.270+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:43.383 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:43.270+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:43.383 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:43.270+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:43.383 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:43.111+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:43.383 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:43.111+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:43.383 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:43.111+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:43.381+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:43.381+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:43.381+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:44.361 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:44.223+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:44.361 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:44.223+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:44.361 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:44.223+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:44.361 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:44.359+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:44.361 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:44.359+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:44.362 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:44.359+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:44.362 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:44.109+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:44.362 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:44.109+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:44.362 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:44.109+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:45.417 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:45.255+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:45.417 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:45.255+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:45.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:45.255+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:45.418 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:45.355+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:45.418 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:45.355+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:45.418 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:45.355+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:45.418 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:45.151+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:45.418 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:45.151+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:45.419 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:45.151+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:46.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:46.175+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:46.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:46.175+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:46.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:46.175+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:46.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:46.209+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:46.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:46.209+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:46.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:46.209+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:46.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:46.357+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:46.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:46.357+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:46.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:46.357+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:47.318+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:47.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:47.318+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:47.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:47.318+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:47.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:47.161+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:47.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:47.161+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:47.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:47.161+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:47.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:47.170+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:47.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:47.170+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:47.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:47.170+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:48.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:48.124+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:48.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:48.124+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:48.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:48.124+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:48.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:48.353+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:48.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:48.353+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:48.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:48.353+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:48.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:48.170+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:48.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:48.170+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:48.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:48.170+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:49.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:49.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:49.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:49.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:49.138+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:49.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:49.138+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:49.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:49.138+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:49.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:49.351+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:49.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:49.351+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:49.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:49.351+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:50.427 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:50.143+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:50.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:50.143+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:50.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:50.143+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:50.428 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:50.377+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:50.428 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:50.377+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:50.428 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:50.377+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:50.428 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:50.178+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:50.429 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:50.178+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:50.429 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:50.178+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:51.183+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:51.183+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:51.183+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:51.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:51.188+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:51.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:51.188+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:51.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:51.188+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:51.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:51.339+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:51.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:51.339+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:51.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:51.339+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:52.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:52.138+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:52.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:52.138+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:52.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:52.138+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:52.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:52.163+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:52.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:52.163+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:52.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:52.163+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:52.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:52.354+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:52.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:52.354+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:52.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:52.354+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:53.371 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:53.119+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:53.371 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:53.119+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:53.371 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:53.119+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:53.372 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:53.183+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:53.372 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:53.183+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:53.372 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:53.183+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:53.370+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:53.370+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:53.370+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:54.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:54.227+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:54.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:54.227+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:54.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:54.227+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:54.080+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:54.080+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:54.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:54.080+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:54.350+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:54.350+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:54.350+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:55.423 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:55.351+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:55.424 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:55.351+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:55.424 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:55.351+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:55.424 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:55.272+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:55.424 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:55.272+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:55.424 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:55.272+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:55.424 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:55.113+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:55.424 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:55.113+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:55.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:55.113+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:56.236+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:56.236+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:56.236+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:56.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:56.099+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:56.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:56.099+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:56.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:56.099+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:56.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:56.342+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:56.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:56.342+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:56.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:56.342+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:57.096+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:57.096+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:57.096+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:57.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:57.296+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:57.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:57.296+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:57.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:57.296+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:57.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:57.250+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:57.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:57.250+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:57.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:57.250+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:58.046+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:58.046+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:58.046+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:58.294+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:58.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:58.294+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:58.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:58.294+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:58.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:58.285+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:58.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:58.285+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:58.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:58.285+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:59.301 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:59.007+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:59.302 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:59.007+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:59.302 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:59.007+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:15:59.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:59.339+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:59.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:59.339+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:59.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:15:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:15:59.339+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:15:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:59.300+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:59.300+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:15:59.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:15:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:15:59.300+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:00.333 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:59.996+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:00.333 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:59.996+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:00.333 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:15:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:15:59.996+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:00.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:00.334+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:00.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:00.334+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:00.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:00.334+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:00.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:00.331+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:00.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:00.331+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:00.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:00.331+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:01.302 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:00.995+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:01.302 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:00.995+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:01.302 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:00.995+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:01.301+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:01.301+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:01.301+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:01.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:01.308+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:01.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:01.308+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:01.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:01.308+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:02.314 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:02.027+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:02.314 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:02.027+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:02.314 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:02.027+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:02.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:02.312+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:02.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:02.312+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:02.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:02.312+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:02.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:02.329+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:02.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:02.329+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:02.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:02.329+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:03.311 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:03.025+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:03.311 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:03.025+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:03.311 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:03.025+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:03.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:03.310+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:03.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:03.310+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:03.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:03.310+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:03.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:03.317+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:03.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:03.317+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:03.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:03.317+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:04.295 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:04.029+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:04.295 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:04.029+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:04.295 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:04.029+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:04.294+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:04.294+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:04.294+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:04.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:04.314+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:04.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:04.314+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:04.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:04.314+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:05.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:05.279+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:05.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:05.279+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:05.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:05.279+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:05.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:05.273+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:05.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:05.273+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:05.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:05.273+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:05.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:05.052+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:05.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:05.052+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:05.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:05.052+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:06.295 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:06.025+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:06.295 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:06.025+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:06.295 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:06.025+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:06.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:06.294+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:06.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:06.294+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:06.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:06.294+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:06.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:06.322+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:06.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:06.322+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:06.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:06.322+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:07.322 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:07.043+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:07.322 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:07.043+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:07.322 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:07.043+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:07.323 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:07.270+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:07.323 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:07.270+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:07.323 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:07.270+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:07.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:07.322+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:07.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:07.322+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:07.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:07.322+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:08.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:08.065+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:08.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:08.065+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:08.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:08.065+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:08.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:08.250+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:08.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:08.250+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:08.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:08.250+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:08.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:08.365+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:08.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:08.365+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:08.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:08.365+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:09.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:09.272+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:09.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:09.272+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:09.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:09.272+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:09.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:09.037+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:09.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:09.037+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:09.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:09.037+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:09.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:09.337+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:09.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:09.338+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:09.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:09.338+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:10.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:10.018+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:10.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:10.018+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:10.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:10.018+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:10.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:10.228+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:10.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:10.228+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:10.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:10.228+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:10.340+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:10.340+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:10.340+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:11.306 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:11.026+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:11.306 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:11.026+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:11.306 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:11.026+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:11.307 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:11.210+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:11.307 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:11.210+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:11.307 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:11.210+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:11.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:11.305+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:11.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:11.305+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:11.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:11.305+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:12.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:11.994+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:12.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:11.994+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:12.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:11.994+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:12.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:12.231+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:12.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:12.231+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:12.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:12.231+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:12.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:12.341+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:12.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:12.341+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:12.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:12.341+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:13.319 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:13.019+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:13.319 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:13.019+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:13.319 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:13.019+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:13.320 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:13.225+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:13.320 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:13.225+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:13.320 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:13.225+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:13.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:13.317+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:13.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:13.317+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:13.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:13.317+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:14.235 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:13.980+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:14.235 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:13.980+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:14.235 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:13.980+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:14.500 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:14.233+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:14.500 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:14.233+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:14.501 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:14.233+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:14.501 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:14.309+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:14.501 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:14.309+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:14.501 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:14.309+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:15.298 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:14.996+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:15.298 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:14.996+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:15.299 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:14.996+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:15.299 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:15.201+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:15.299 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:15.201+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:15.299 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:15.201+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:15.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:15.297+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:15.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:15.297+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:15.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:15.297+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:16.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:16.031+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:16.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:16.031+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:16.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:16.031+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:16.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:16.244+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:16.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:16.244+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:16.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:16.244+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:16.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:16.337+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:16.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:16.337+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:16.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:16.337+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:17.260 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:16.981+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:17.260 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:16.981+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:17.260 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:16.981+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:17.372+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:17.372+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:17.372+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:17.258+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:17.258+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:17.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:17.258+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:18.335 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:17.993+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:18.335 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:17.993+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:18.335 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:17.993+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:18.335 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:18.228+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:18.335 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:18.228+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:18.335 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:18.228+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:18.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:18.333+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:18.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:18.333+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:18.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:18.333+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:19.292 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:19.020+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:19.292 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:19.020+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:19.293 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:19.020+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:19.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:19.217+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:19.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:19.217+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:19.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:19.217+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:19.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:19.290+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:19.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:19.290+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:19.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:19.291+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:20.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:20.204+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:20.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:20.204+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:20.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:20.204+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:20.026+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:20.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:20.026+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:20.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:20.026+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:20.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:20.243+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:20.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:20.243+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:20.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:20.243+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:21.254 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:21.056+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:21.254 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:21.056+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:21.254 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:21.056+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:21.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:21.268+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:21.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:21.268+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:21.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:21.268+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:21.252+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:21.252+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:21.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:21.252+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:22.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:22.269+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:22.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:22.269+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:22.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:22.269+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:22.054+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:22.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:22.054+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:22.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:22.054+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:22.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:22.289+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:22.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:22.289+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:22.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:22.289+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:23.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:23.265+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:23.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:23.265+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:23.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:23.265+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:23.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:23.073+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:23.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:23.073+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:23.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:23.073+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:23.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:23.313+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:23.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:23.313+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:23.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:23.313+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:24.341 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:24.285+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:24.341 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:24.285+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:24.342 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:24.285+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:24.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:24.082+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:24.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:24.082+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:24.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:24.082+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:24.342 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:24.331+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:24.342 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:24.331+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:24.342 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:24.331+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:25.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:25.304+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:25.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:25.304+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:25.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:25.304+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:25.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:25.275+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:25.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:25.275+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:25.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:25.275+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:25.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:25.054+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:25.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:25.054+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:25.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:25.054+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:26.337 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:26.036+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:26.337 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:26.036+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:26.337 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:26.036+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:26.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:26.281+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:26.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:26.281+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:26.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:26.281+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:26.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:26.336+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:26.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:26.336+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:26.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:26.336+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:27.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:27.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:27.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:27.062+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:27.062+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:27.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:27.062+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:27.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:27.363+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:27.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:27.363+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:27.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:27.363+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:28.084+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:28.084+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:28.084+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:28.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:28.201+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:28.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:28.201+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:28.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:28.201+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:28.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:28.390+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:28.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:28.390+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:28.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:28.390+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:29.399 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:29.133+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:29.400 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:29.133+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:29.400 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:29.133+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:29.400 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:29.218+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:29.400 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:29.218+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:29.400 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:29.218+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:29.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:29.398+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:29.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:29.398+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:29.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:29.398+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:30.430 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:30.158+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:30.430 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:30.158+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:30.430 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:30.158+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:30.431 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:30.370+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:30.431 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:30.370+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:30.431 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:30.370+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:30.431 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:30.243+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:30.431 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:30.243+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:30.431 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:30.243+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:31.395 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:31.127+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:31.395 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:31.127+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:31.395 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:31.127+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:31.395 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:31.287+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:31.395 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:31.287+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:31.395 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:31.287+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:31.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:31.393+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:31.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:31.393+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:31.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:31.393+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:32.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:32.084+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:32.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:32.084+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:32.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:32.084+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:32.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:32.260+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:32.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:32.260+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:32.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:32.260+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:32.349+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:32.349+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:32.349+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:33.222+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:33.222+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:33.222+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:33.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:33.043+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:33.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:33.043+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:33.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:33.043+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:33.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:33.382+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:33.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:33.382+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:33.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:33.382+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:34.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:34.015+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:34.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:34.015+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:34.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:34.015+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:34.233+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:34.234+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:34.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:34.234+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:34.355+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:34.355+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:34.355+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:35.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:34.989+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:35.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:34.989+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:35.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:34.989+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:35.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:35.232+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:35.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:35.232+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:35.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:35.232+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:35.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:35.394+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:35.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:35.394+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:35.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:35.394+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:36.005+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:36.005+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:36.005+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:36.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:36.200+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:36.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:36.200+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:36.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:36.200+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:36.406+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:36.406+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:36.406+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:37.046+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:37.046+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:37.046+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:37.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:37.203+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:37.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:37.203+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:37.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:37.203+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:37.403+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:37.403+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:37.403+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:38.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:38.042+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:38.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:38.042+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:38.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:38.042+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:38.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:38.179+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:38.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:38.179+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:38.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:38.179+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:38.373+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:38.373+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:38.373+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:39.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:39.152+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:39.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:39.152+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:39.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:39.152+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:39.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:39.002+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:39.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:39.002+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:39.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:39.002+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:39.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:39.348+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:39.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:39.348+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:39.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:39.348+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:40.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:40.025+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:40.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:40.025+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:40.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:40.025+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:40.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:40.113+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:40.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:40.113+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:40.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:40.113+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:40.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:40.390+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:40.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:40.390+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:40.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:40.390+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:41.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:41.028+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:41.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:41.028+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:41.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:41.028+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:41.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:41.131+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:41.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:41.131+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:41.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:41.131+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:41.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:41.439+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:41.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:41.439+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:41.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:41.439+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:42.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:42.149+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:42.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:42.149+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:42.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:42.149+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:42.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:41.992+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:42.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:41.992+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:42.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:41.992+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:42.473+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:42.474+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:42.474+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:43.004+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:43.004+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:43.004+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:43.147+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:43.147+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:43.147+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:43.490+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:43.490+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:43.490+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:43.990+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:43.990+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:43.990+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:44.192+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:44.192+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:44.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:44.192+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:44.466+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:44.466+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:44.466+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:45.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:44.952+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:45.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:44.952+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:45.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:44.952+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:45.543 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:45.426+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:45.543 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:45.426+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:45.543 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:45.426+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:45.543 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:45.228+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:45.544 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:45.228+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:45.544 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:45.228+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:46.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:45.944+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:46.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:45.944+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:46.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:45.944+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:46.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:46.376+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:46.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:46.376+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:46.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:46.376+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:46.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:46.224+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:46.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:46.224+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:46.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:46.224+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:47.235 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:46.958+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:47.238 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:46.958+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:47.238 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:46.958+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:47.368+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:47.368+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:47.368+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:47.234+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:47.234+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:47.234+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:48.191 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:47.914+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:48.191 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:47.914+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:48.191 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:47.914+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:48.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:48.391+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:48.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:48.391+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:48.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:48.391+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:48.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:48.190+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:48.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:48.190+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:48.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:48.190+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:49.233 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:48.943+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:49.233 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:48.943+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:49.233 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:48.943+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:49.431+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:49.431+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:49.431+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:49.232+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:49.232+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:49.232+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:50.267 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:49.969+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:50.267 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:49.969+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:50.267 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:49.969+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:50.544 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:50.452+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:50.544 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:50.452+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:50.544 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:50.452+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:50.544 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:50.266+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:50.544 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:50.266+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:50.544 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:50.266+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:51.257 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:50.978+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:51.257 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:50.979+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:51.257 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:50.979+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:51.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:51.496+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:51.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:51.496+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:51.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:51.496+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:51.255+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:51.255+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:51.255+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:52.248 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:51.992+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:52.249 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:51.992+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:52.249 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:51.992+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:52.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:52.247+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:52.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:52.247+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:52.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:52.247+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:52.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:52.472+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:52.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:52.472+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:52.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:52.472+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:53.292 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:52.987+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:53.293 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:52.987+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:53.293 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:52.987+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:53.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:53.291+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:53.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:53.291+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:53.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:53.291+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:53.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:53.435+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:53.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:53.435+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:53.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:53.435+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:54.015+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:54.015+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:54.015+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:54.429+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:54.429+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:54.429+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:54.340+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:54.340+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:54.340+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:55.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:55.016+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:55.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:55.016+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:55.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:55.016+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:55.356+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:55.356+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:55.356+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:55.421+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:55.421+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:55.421+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:56.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:56.055+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:56.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:56.055+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:56.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:56.055+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:56.391+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:56.391+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:56.391+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:56.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:56.346+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:56.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:56.346+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:56.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:56.346+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:57.314 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:57.020+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:57.314 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:57.020+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:57.314 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:57.020+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:57.312+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:57.312+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:57.312+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:57.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:57.381+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:57.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:57.381+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:57.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:57.381+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:58.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:57.993+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:58.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:57.993+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:58.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:57.993+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:58.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:58.333+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:58.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:58.333+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:58.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:58.333+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:58.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:58.355+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:58.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:58.355+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:58.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:58.355+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:59.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:58.981+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:59.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:58.981+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:59.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:58.981+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:16:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:59.303+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:59.303+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:16:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:16:59.303+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:16:59.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:59.350+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:59.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:59.350+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:16:59.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:16:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:16:59.350+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:00.258 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:59.936+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:00.258 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:59.936+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:00.258 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:16:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:16:59.936+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:00.545 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:00.320+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:00.545 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:00.320+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:00.545 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:00.320+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:00.545 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:00.257+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:00.545 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:00.257+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:00.545 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:00.257+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:01.298 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:00.929+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:01.298 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:00.929+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:01.299 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:00.929+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:01.297+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:01.297+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:01.297+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:01.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:01.340+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:01.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:01.340+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:01.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:01.340+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:01.941+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:01.941+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:01.941+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:02.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:02.344+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:02.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:02.344+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:02.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:02.344+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:02.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:02.369+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:02.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:02.369+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:02.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:02.369+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:03.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:02.897+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:03.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:02.897+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:03.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:02.897+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:03.652 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:03.411+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:03.652 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:03.411+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:03.652 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:03.411+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:03.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:03.351+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:03.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:03.351+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:03.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:03.351+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:04.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:03.850+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:04.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:03.850+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:04.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:03.850+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:04.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:04.455+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:04.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:04.455+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:04.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:04.456+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:04.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:04.367+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:04.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:04.367+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:04.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:04.367+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:04.872+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:04.872+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:04.872+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:05.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:05.470+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:05.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:05.470+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:05.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:05.470+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:05.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:05.400+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:05.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:05.400+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:05.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:05.400+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:06.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:05.858+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:06.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:05.858+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:06.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:05.858+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:06.726 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:06.507+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:06.726 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:06.507+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:06.726 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:06.507+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:06.727 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:06.431+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:06.727 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:06.431+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:06.727 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:06.431+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:07.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:06.813+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:07.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:06.813+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:07.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:06.813+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:07.777 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:07.550+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:07.777 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:07.550+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:07.777 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:07.550+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:07.777 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:07.429+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:07.777 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:07.429+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:07.777 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:07.429+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:08.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:07.775+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:08.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:07.775+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:08.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:07.775+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:08.744 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:08.502+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:08.744 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:08.502+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:08.745 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:08.502+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:08.745 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:08.444+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:08.745 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:08.444+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:08.745 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:08.444+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:09.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:08.743+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:09.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:08.743+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:09.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:08.743+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:09.531 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:09.477+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:09.531 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:09.477+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:09.531 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:09.477+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:09.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:09.720+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:09.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:09.720+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:09.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:09.720+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:09.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:09.529+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:09.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:09.529+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:09.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:09.529+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:10.487+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:10.487+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:10.487+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:10.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:10.671+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:10.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:10.671+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:10.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:10.671+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:10.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:10.468+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:10.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:10.468+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:10.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:10.468+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:11.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:11.477+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:11.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:11.477+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:11.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:11.477+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:11.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:11.640+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:11.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:11.640+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:11.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:11.640+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:11.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:11.480+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:11.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:11.480+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:11.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:11.480+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:12.448+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:12.448+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:12.448+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:12.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:12.440+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:12.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:12.440+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:12.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:12.440+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:12.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:12.591+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:12.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:12.591+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:12.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:12.591+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:13.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:13.597+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:13.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:13.597+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:13.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:13.597+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:13.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:13.479+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:13.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:13.479+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:13.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:13.479+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:13.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:13.419+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:13.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:13.419+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:13.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:13.419+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:14.405+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:14.405+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:14.405+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:14.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:14.503+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:14.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:14.503+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:14.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:14.503+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:14.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:14.555+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:14.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:14.555+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:14.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:14.555+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:15.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:17:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:17:15.422+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:17:15.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:15.592+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:15.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:15.592+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:15.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:15.592+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:15.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:15.521+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:15.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:15.521+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:15.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:15.521+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:15.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:15.397+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:15.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:15.397+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:15.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:15.397+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:16.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:16.543+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:16.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:16.543+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:16.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:16.543+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:16.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:16.533+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:16.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:16.533+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:16.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:16.533+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:16.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:16.406+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:16.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:16.406+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:16.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:16.406+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:17.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:17.535+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:17.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:17.535+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:17.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:17.535+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:17.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:17.511+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:17.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:17.511+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:17.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:17.511+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:17.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:17.357+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:17.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:17.357+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:17.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:17.357+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:18.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:18.559+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:18.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:18.559+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:18.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:18.559+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:18.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:18.386+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:18.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:18.386+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:18.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:18.386+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:18.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:18.578+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:18.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:18.578+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:18.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:18.578+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:19.558 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:19.408+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:19.559 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:19.408+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:19.559 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:19.408+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:19.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:19.566+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:19.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:19.566+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:19.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:19.566+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:19.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:19.557+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:19.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:19.557+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:19.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:19.557+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:20.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:17:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:17:20.422+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:17:20.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:20.610+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:20.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:20.610+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:20.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:20.610+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:20.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:20.387+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:20.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:20.387+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:20.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:20.387+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:20.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:20.582+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:20.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:20.582+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:20.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:20.582+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:21.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:21.592+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:21.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:21.592+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:21.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:21.592+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:21.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:21.390+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:21.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:21.390+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:21.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:21.390+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:21.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:21.550+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:21.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:21.550+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:21.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:21.550+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:22.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:22.580+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:22.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:22.580+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:22.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:22.580+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:22.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:22.399+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:22.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:22.399+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:22.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:22.399+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:22.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:22.584+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:22.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:22.584+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:22.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:22.584+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:23.433+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:23.433+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:23.433+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:23.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:23.597+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:23.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:23.597+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:23.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:23.597+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:23.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:23.613+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:23.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:23.613+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:23.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:23.613+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:24.659 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:24.581+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:24.659 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:24.581+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:24.659 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:24.581+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:24.660 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:24.408+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:24.660 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:24.408+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:24.660 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:24.408+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:25.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:24.658+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:25.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:24.659+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:25.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:24.659+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:25.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:17:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:17:25.423+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:17:25.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:25.395+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:25.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:25.395+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:25.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:25.395+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:25.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:25.569+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:25.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:25.569+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:25.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:25.569+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:26.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:25.699+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:26.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:25.699+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:26.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:25.699+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:26.715 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:26.555+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:26.715 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:26.555+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:26.715 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:26.555+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:26.716 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:26.349+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:26.716 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:26.349+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:26.716 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:26.349+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:27.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:26.714+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:27.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:26.714+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:27.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:26.714+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:27.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:27.356+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:27.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:27.356+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:27.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:27.356+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:27.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:27.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:27.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:27.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:27.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:27.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:28.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:27.701+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:28.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:27.701+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:28.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:27.701+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:28.529 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:28.337+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:28.529 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:28.337+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:28.529 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:28.337+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:28.674+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:28.674+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:28.674+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:28.529+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:28.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:28.529+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:28.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:28.529+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:29.531 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:29.345+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:29.531 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:29.345+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:29.531 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:29.345+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:29.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:29.681+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:29.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:29.681+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:29.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:29.681+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:29.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:29.529+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:29.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:29.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:29.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:29.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:30.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:17:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:17:30.423+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 36 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:17:30.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:30.543+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:30.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:30.543+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:30.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:30.543+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:30.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:30.335+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:30.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:30.335+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:30.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:30.335+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:31.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:30.644+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:31.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:30.644+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:31.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:30.644+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:31.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:31.382+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:31.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:31.382+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:31.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:31.382+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:31.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:31.599+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:31.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:31.599+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:31.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:31.599+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:31.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:31.522+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:31.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:31.522+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:31.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:31.522+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:32.381+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:32.381+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:32.381+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:32.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:32.519+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:32.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:32.519+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:32.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:32.519+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:32.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:32.598+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:32.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:32.598+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:32.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:32.598+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:33.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:33.551+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:33.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:33.551+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:33.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:33.551+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:33.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:33.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:33.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:33.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:33.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:33.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:33.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:33.372+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:33.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:33.372+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:33.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:33.372+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:34.591+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:34.591+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:34.591+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:34.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:34.586+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:34.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:34.586+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:34.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:34.586+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:34.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:34.379+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:34.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:34.379+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:34.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:34.379+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:35.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:17:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:17:35.423+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 68 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:17:35.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:35.562+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:35.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:35.562+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:35.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:35.562+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:35.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:35.410+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:35.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:35.410+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:35.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:35.410+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:35.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:35.565+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:35.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:35.565+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:35.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:35.565+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:36.550+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:36.550+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:36.550+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:36.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:36.599+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:36.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:36.599+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:36.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:36.599+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:36.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:36.413+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:36.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:36.413+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:36.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:36.413+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:37.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:37.554+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:37.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:37.554+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:37.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:37.554+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:37.626+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:37.626+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:37.626+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:37.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:37.460+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:37.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:37.460+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:37.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:37.460+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:38.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:38.547+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:38.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:38.547+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:38.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:38.547+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:38.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:38.592+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:38.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:38.592+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:38.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:38.592+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:38.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:38.502+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:38.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:38.502+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:38.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:38.502+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:39.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:39.549+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:39.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:39.549+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:39.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:39.549+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:39.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:39.542+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:39.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:39.542+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:39.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:39.542+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:39.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:39.522+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:39.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:39.522+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:39.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:39.522+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:40.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:17:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:17:40.424+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 98 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:17:40.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:40.574+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:40.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:40.574+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:40.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:40.574+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:40.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:40.478+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:40.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:40.478+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:40.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:40.478+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:40.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:40.523+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:40.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:40.523+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:40.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:40.523+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:41.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:41.618+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:41.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:41.618+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:41.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:41.618+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:41.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:41.541+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:41.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:41.541+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:41.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:41.541+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:41.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:41.503+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:41.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:41.503+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:41.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:41.503+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:42.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:42.529+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:42.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:42.529+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:42.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:42.529+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:42.635+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:42.635+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:42.635+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:42.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:42.478+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:42.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:42.478+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:42.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:42.478+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:43.591+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:43.591+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:43.591+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:43.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:43.557+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:43.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:43.557+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:43.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:43.557+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:43.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:43.504+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:43.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:43.504+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:43.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:43.504+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:44.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:44.545+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:44.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:44.545+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:44.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:44.545+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:44.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:44.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:44.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:44.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:44.473+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:44.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:44.473+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:44.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:44.473+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:45.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:17:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:17:45.424+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 127 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:17:45.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:45.461+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:45.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:45.461+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:45.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:45.461+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:45.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:45.556+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:45.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:45.556+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:45.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:45.556+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:45.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:45.577+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:45.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:45.577+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:45.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:45.577+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:46.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:46.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:46.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:46.597+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:46.597+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:46.597+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:46.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:46.485+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:46.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:46.485+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:46.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:46.485+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:47.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:47.617+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:47.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:47.617+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:47.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:47.617+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:47.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:47.609+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:47.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:47.609+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:47.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:47.609+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:47.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:47.468+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:47.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:47.468+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:47.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:47.468+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:48.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:48.451+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:48.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:48.451+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:48.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:48.451+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:48.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:48.655+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:48.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:48.655+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:48.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:48.655+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:48.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:48.594+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:48.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:48.594+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:48.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:48.594+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:49.418+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:49.418+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:49.418+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:49.605+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:49.605+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:49.605+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:49.688+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:49.688+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:49.688+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:50.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:17:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:17:50.425+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 156 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:17:50.637 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:50.371+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:50.637 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:50.371+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:50.638 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:50.371+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:50.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:50.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:50.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:51.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:50.636+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:51.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:50.636+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:51.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:50.636+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:51.629 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:51.345+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:51.629 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:51.346+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:51.629 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:51.346+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:51.654+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:51.654+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:51.654+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:51.670+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:51.670+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:51.670+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:52.647 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:52.387+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:52.647 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:52.387+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:52.647 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:52.387+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:53.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:52.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:53.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:52.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:53.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:52.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:53.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:52.645+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:53.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:52.645+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:53.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:52.645+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:53.693 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:53.608+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:53.694 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:53.608+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:53.694 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:53.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:53.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:53.371+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:53.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:53.371+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:53.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:53.371+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:54.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:53.691+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:54.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:53.692+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:54.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:53.692+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:54.651 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:54.396+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:54.651 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:54.396+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:54.651 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:54.396+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:55.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:54.656+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:55.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:54.657+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:55.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:54.657+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:55.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:54.650+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:55.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:54.650+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:55.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:54.650+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:55.695 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:17:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:17:55.425+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 185 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:17:55.696 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:55.618+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:55.696 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:55.618+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:55.696 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:55.618+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:55.696 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:55.399+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:55.696 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:55.399+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:55.696 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:55.399+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:56.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:55.694+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:56.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:55.694+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:56.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:55.694+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:56.678 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:56.610+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:56.678 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:56.610+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:56.678 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:56.610+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:56.678 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:56.367+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:56.678 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:56.367+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:56.678 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:56.367+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:57.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:56.676+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:57.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:56.676+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:57.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:56.676+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:57.568+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:57.568+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:57.568+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:57.331+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:57.331+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:57.331+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:58.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:57.648+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:58.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:57.648+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:58.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:57.648+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:58.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:58.324+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:58.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:58.324+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:58.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:58.324+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:58.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:58.610+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:58.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:58.610+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:58.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:58.610+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:17:58.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:58.693+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:58.693+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:58.693+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:17:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:59.340+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:59.340+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:17:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:17:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:17:59.340+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:59.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:59.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:17:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:17:59.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:00.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:59.650+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:00.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:59.650+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:00.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:17:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:17:59.650+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:00.691 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:18:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:18:00.426+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 212 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:18:00.691 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:00.372+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:00.691 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:00.372+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:00.692 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:00.372+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:00.692 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:00.601+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:00.692 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:00.601+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:00.692 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:00.601+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:00.689+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:00.689+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:00.689+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:01.630 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:01.378+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:01.630 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:01.378+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:01.630 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:01.378+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:01.630 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:01.593+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:01.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:01.593+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:01.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:01.593+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:02.020 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:01.667+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:02.020 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:01.667+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:02.020 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:01.667+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:02.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:02.339+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:02.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:02.339+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:02.654 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:02.339+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:02.654 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:02.583+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:02.654 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:02.583+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:02.654 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:02.583+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:02.652+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:02.652+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:02.652+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:03.621 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:03.568+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:03.621 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:03.568+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:03.621 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:03.568+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:03.621 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:03.354+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:03.622 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:03.354+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:03.622 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:03.354+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:04.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:03.620+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:04.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:03.620+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:04.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:03.620+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:04.319+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:04.319+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:04.319+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:04.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:04.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:04.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:05.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:04.617+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:05.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:04.617+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:05.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:04.617+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:05.428 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:05.340+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:05.428 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:05.340+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:05.428 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:05.340+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:05.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:18:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:18:05.426+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 241 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:18:05.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:05.572+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:05.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:05.572+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:05.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:05.572+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:05.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:05.662+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:05.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:05.662+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:05.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:05.662+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:06.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:06.330+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:06.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:06.330+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:06.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:06.330+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:07.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:06.704+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:07.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:06.704+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:07.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:06.704+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:06.593+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:06.593+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:06.593+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:07.608 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:07.355+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:07.608 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:07.355+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:07.608 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:07.355+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:08.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:07.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:08.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:07.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:08.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:07.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:07.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:07.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:07.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:08.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:08.313+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:08.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:08.313+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:08.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:08.313+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:09.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:08.625+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:09.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:08.625+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:09.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:08.625+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:08.609+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:08.609+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:08.609+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:09.581 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:09.271+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:09.581 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:09.271+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:09.582 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:09.271+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:09.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:09.580+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:09.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:09.580+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:09.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:09.580+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:09.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:09.648+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:09.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:09.648+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:09.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:09.648+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:10.549 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:18:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:18:10.427+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 268 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:18:10.550 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:10.264+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:10.550 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:10.264+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:10.550 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:10.264+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:10.611+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:10.611+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:10.611+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:10.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:10.564+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:10.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:10.564+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:10.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:10.564+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:11.573 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:11.260+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:11.573 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:11.260+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:11.573 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:11.260+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:11.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:11.570+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:11.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:11.570+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:11.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:11.570+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:11.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:11.570+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:11.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:11.570+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:11.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:11.570+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:12.522 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:12.232+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:12.522 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:12.232+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:12.522 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:12.232+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:12.555+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:12.555+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:12.555+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:12.521+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:12.521+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:12.521+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:13.518 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:13.496+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:13.519 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:13.496+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:13.519 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:13.496+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:13.519 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:13.251+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:13.519 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:13.251+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:13.519 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:13.251+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:13.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:13.593+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:13.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:13.593+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:13.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:13.593+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:14.457+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:14.457+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:14.457+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:14.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:14.295+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:14.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:14.295+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:14.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:14.295+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:14.636+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:14.636+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:14.636+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:15.684 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:18:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:18:15.427+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 301 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:18:15.684 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:15.340+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:15.684 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:15.340+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:15.684 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:15.340+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:15.685 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:15.453+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:15.685 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:15.453+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:15.685 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:15.453+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:15.683+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:15.683+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:15.683+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:16.499+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:16.499+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:16.499+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:16.319+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:16.319+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:16.319+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:16.697+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:16.697+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:16.697+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:17.517+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:17.517+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:17.517+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:17.292+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:17.292+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:17.292+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:17.649+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:17.649+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:17.649+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:18.536+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:18.536+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:18.536+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:18.537+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:18.328+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:18.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:18.328+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:18.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:18.328+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:18.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:18.328+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14530.0:4601 3.1e 3:7aab642c:::10000003cc7.00000000:head [write 0~215 in=215b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:18.638+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:18.638+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:18.638+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:18.639+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:19.576 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:19.293+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:19.576 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:19.293+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:19.576 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:19.293+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:19.577 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:19.294+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:19.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:19.575+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:19.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:19.575+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:19.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:19.575+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:19.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:19.575+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:19.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:19.683+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:19.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:19.683+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:19.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:19.683+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:19.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:19.684+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:20.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:18:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:18:20.428+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 328 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:18:20.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:20.321+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:20.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:20.321+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:20.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:20.321+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:20.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:20.322+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:20.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:20.548+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:20.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:20.548+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:20.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:20.548+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:20.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:20.549+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:21.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:20.702+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:21.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:20.702+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:21.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:20.702+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:21.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:20.702+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:21.334+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:21.334+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:21.334+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:21.335+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:21.505+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:21.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:21.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:21.506+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:22.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:21.680+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:22.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:21.680+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:22.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:21.680+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:22.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:21.681+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:22.727 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:22.367+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:22.728 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:22.367+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:22.728 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:22.367+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:22.728 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:22.368+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:22.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:22.470+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:22.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:22.470+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:22.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:22.470+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:22.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:22.470+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:22.726+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:22.726+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:22.726+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:22.727+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:23.766 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:23.509+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:23.766 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:23.509+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:23.766 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:23.509+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:23.766 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:23.511+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:23.766 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:23.386+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:23.766 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:23.386+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:23.767 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:23.386+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:23.767 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:23.387+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:24.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:23.764+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:24.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:23.764+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:24.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:23.764+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:24.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:23.765+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:24.735 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:24.417+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:24.735 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:24.417+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:24.735 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:24.417+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:24.736 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:24.417+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:24.736 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:24.504+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:24.736 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:24.504+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:24.736 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:24.504+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:24.736 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:24.505+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:25.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:24.734+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:25.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:24.734+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:25.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:24.734+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:25.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:24.734+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:25.771 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:18:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:18:25.428+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 362 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:18:25.771 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:25.508+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:25.772 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:25.508+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:25.772 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:25.508+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:25.772 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:25.509+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:25.772 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:25.452+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:25.772 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:25.452+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:25.772 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:25.452+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:25.772 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:25.453+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:26.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:25.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:26.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:25.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:26.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:25.770+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:26.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:25.771+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:26.797 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:26.451+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:26.797 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:26.451+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:26.797 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:26.451+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:26.797 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:26.452+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:26.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:26.501+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:26.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:26.501+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:26.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:26.501+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:26.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:26.501+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:27.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:26.795+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:27.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:26.795+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:27.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:26.795+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:27.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:26.796+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:27.801 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:27.538+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:27.801 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:27.538+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:27.802 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:27.538+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:27.802 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:27.539+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:27.802 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:27.443+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:27.802 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:27.443+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:27.802 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:27.443+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:27.802 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:27.444+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:28.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:27.800+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:28.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:27.800+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:28.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:27.800+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:28.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:27.801+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:28.762 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:28.504+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:28.762 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:28.504+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:28.762 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:28.504+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:28.762 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:28.504+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:28.762 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:28.458+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:28.762 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:28.458+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:28.763 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:28.458+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:28.763 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:28.459+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:28.760+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:28.760+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:28.760+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:28.761+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:29.506 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:29.489+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:29.506 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:29.489+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:29.506 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:29.489+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:29.507 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:29.489+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:29.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:29.505+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:29.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:29.505+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:29.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:29.505+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:29.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:29.506+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:29.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:29.719+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:29.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:29.719+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:29.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:29.719+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:29.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:29.719+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:30.733 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:18:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:18:30.429+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 393 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:18:30.733 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:30.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:30.733 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:30.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:30.733 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:30.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:30.733 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:30.531+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:30.733 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:30.482+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:30.734 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:30.482+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:30.734 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:30.482+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:30.734 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:30.482+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:30.731+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:30.731+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:30.731+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:30.732+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:31.766 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:31.522+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:31.766 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:31.522+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:31.766 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:31.522+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:31.766 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:31.522+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:31.766 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:31.509+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:31.766 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:31.509+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:31.766 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:31.509+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:31.766 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:31.510+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:31.764+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:31.764+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:31.764+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:31.765+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:32.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:32.517+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:32.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:32.517+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:32.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:32.517+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:32.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:32.518+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:32.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:32.730+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:32.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:32.730+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:32.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:32.730+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:32.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:32.730+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:32.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:32.554+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:32.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:32.554+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:32.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:32.554+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:32.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:32.555+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:33.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:33.524+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:33.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:33.524+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:33.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:33.524+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:33.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:33.525+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:33.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:33.723+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:33.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:33.723+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:33.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:33.723+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:33.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:33.724+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:33.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:33.556+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:33.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:33.556+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:33.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:33.556+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:33.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:33.557+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:34.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:34.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:34.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:34.742+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:34.538+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:34.538+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:34.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:34.538+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:34.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:34.538+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:34.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:34.548+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:34.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:34.548+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:34.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:34.548+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:34.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:34.548+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:35.735 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:18:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:18:35.429+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 422 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:18:35.735 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:35.570+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:35.736 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:35.570+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:35.736 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:35.570+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:35.736 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:35.570+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:35.736 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:35.538+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:35.736 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:35.538+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:35.736 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:35.538+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:35.736 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:35.538+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:35.733+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:35.733+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:35.733+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:35.734+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:36.713+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:36.713+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:36.713+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:36.714+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:36.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:36.586+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:36.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:36.586+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:36.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:36.586+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:36.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:36.586+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:36.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:36.496+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:36.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:36.496+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:36.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:36.496+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:36.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:36.496+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:37.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:37.460+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:37.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:37.460+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:37.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:37.460+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:37.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:37.462+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:37.581+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:37.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:37.581+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:37.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:37.581+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:37.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:37.581+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:37.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:37.666+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:37.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:37.666+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:37.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:37.666+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:37.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:37.666+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:38.619+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:38.619+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:38.619+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:38.620+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:38.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:38.611+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:38.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:38.611+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:38.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:38.611+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:38.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:38.612+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:38.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:38.473+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:38.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:38.473+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:38.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:38.473+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:38.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:38.474+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:39.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:39.447+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:39.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:39.447+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:39.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:39.447+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:39.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:39.448+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:40.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:39.599+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:40.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:39.599+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:40.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:39.599+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:40.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:39.600+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:39.605+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:40.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:39.605+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:40.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:39.605+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:40.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:39.605+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:40.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:18:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:18:40.431+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 449 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:18:40.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:40.404+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:40.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:40.404+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:40.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:40.404+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:40.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:40.404+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:40.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:40.570+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:40.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:40.570+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:40.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:40.570+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:40.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:40.572+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:40.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:40.552+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:40.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:40.552+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:40.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:40.552+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:40.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:40.552+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:41.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:41.527+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:41.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:41.527+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:41.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:41.527+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:41.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:41.527+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:41.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:41.562+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:41.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:41.562+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:41.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:41.562+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:41.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:41.563+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:41.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:41.439+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:41.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:41.439+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:41.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:41.439+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:41.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:41.440+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:42.533+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:42.533+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:42.533+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:42.533+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:42.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:42.390+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:42.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:42.390+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:42.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:42.390+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:42.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:42.391+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:42.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:42.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:42.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:42.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:42.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:42.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:42.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:42.572+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:43.424+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:43.424+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:43.424+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:43.425+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:43.578+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:43.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:43.578+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:43.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:43.578+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:43.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:43.578+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:43.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:43.554+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:43.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:43.554+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:43.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:43.554+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:43.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:43.555+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:44.564+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:44.564+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:44.564+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:44.565+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:44.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:44.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:44.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:44.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:44.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:44.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:44.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:44.510+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:44.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:44.377+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:44.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:44.377+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:44.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:44.377+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:44.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:44.378+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:45.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:18:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:18:45.431+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 483 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:18:45.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:45.575+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:45.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:45.575+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:45.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:45.575+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:45.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:45.576+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:45.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:45.416+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:45.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:45.416+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:45.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:45.416+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:45.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:45.416+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:45.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:45.531+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:45.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:45.531+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:45.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:45.531+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:45.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:45.532+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:46.550+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:46.550+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:46.550+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:46.551+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:46.567+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:46.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:46.567+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:46.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:46.567+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:46.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:46.568+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:46.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:46.380+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:46.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:46.380+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:46.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:46.380+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:46.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:46.380+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:47.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:47.507+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:47.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:47.508+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:47.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:47.508+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:47.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:47.508+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:47.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:47.346+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:47.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:47.346+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:47.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:47.346+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:47.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:47.347+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:47.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:47.584+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:47.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:47.584+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:47.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:47.584+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:47.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:47.584+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:48.551 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:48.532+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:48.551 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:48.532+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:48.551 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:48.532+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:48.551 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:48.533+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:48.552 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:48.298+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:48.552 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:48.299+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:48.552 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:48.299+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:48.552 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:48.299+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:48.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:48.549+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:48.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:48.549+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:48.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:48.549+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:48.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:48.551+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:49.518 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:49.279+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:49.518 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:49.279+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:49.518 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:49.279+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:49.518 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:49.280+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:49.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:49.516+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:49.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:49.516+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:49.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:49.516+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:49.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:49.517+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:49.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:49.586+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:49.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:49.586+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:49.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:49.586+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:49.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:49.587+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:50.557 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:18:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:18:50.430+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 514 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:18:50.557 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:50.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:50.557 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:50.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:50.557 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:50.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:50.557 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:50.239+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:50.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:50.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:50.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:50.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:50.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:50.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:50.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:50.567+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:50.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:50.619+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:50.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:50.619+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:50.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:50.619+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:50.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:50.620+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:51.583 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:51.209+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:51.583 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:51.209+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:51.583 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:51.209+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:51.583 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:51.209+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:51.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:51.581+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:51.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:51.581+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:51.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:51.581+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:51.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:51.582+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:51.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:51.594+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:51.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:51.594+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:51.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:51.594+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:51.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:51.595+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:52.573 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:52.209+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:52.573 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:52.209+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:52.573 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:52.209+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:52.573 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:52.209+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:52.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:52.625+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:52.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:52.625+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:52.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:52.625+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:52.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:52.625+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:52.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:52.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:52.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:52.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:52.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:52.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:52.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:52.572+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:53.521 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:53.179+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:53.521 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:53.179+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:53.521 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:53.179+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:53.521 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:53.180+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:53.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:53.560+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:53.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:53.560+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:53.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:53.560+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:53.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:53.561+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:53.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:53.650+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:53.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:53.650+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:53.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:53.650+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:53.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:53.651+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:54.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:54.148+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:54.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:54.148+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:54.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:54.148+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:54.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:54.149+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:54.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:54.550+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:54.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:54.550+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:54.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:54.550+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:54.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:54.550+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:54.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:54.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:54.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:54.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:54.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:54.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:54.617+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:55.433 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:55.157+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:55.433 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:55.157+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:55.433 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:55.157+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:55.433 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:55.158+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:55.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:18:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:18:55.431+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 543 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:18:55.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:55.526+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:55.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:55.526+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:55.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:55.526+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:55.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:55.527+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:55.577+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:55.577+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:55.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:55.577+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:55.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:55.577+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:56.484 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:56.109+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:56.484 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:56.109+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:56.484 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:56.109+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:56.484 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:56.110+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:56.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:56.483+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:56.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:56.483+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:56.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:56.483+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:56.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:56.483+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:56.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:56.609+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:56.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:56.609+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:56.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:56.609+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:56.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:56.610+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:57.465 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:57.113+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:57.465 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:57.113+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:57.465 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:57.113+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:57.465 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:57.114+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:57.463+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:57.463+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:57.463+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:57.464+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:57.570+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:57.570+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:57.570+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:57.571+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:58.426 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:58.154+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:58.426 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:58.154+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:58.426 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:58.154+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:58.426 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:58.154+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:58.424+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:58.424+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:58.424+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:58.425+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:58.598+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:58.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:58.598+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:58.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:58.598+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:58.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:58.599+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:59.396 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:59.144+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:59.396 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:59.144+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:59.396 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:59.144+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:18:59.396 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:18:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:18:59.145+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:18:59.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:59.585+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:59.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:59.585+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:59.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:59.585+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:18:59.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:18:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:18:59.585+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:18:59.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:59.394+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:59.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:59.394+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:59.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:59.394+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:18:59.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:18:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:18:59.395+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:00.439 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:19:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:19:00.431+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 570 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:19:00.439 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:00.369+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:00.439 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:00.369+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:00.439 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:00.369+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:00.439 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:00.369+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:00.440 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:00.183+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:00.440 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:00.183+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:00.440 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:00.183+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:00.440 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:00.184+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:00.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:00.618+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:00.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:00.618+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:00.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:00.618+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:00.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:00.619+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:01.147+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:01.147+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:01.147+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:01.148+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:01.334+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:01.334+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:01.334+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:01.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:01.334+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:01.637+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:01.637+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:01.637+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:02.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:01.638+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:02.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:02.122+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:02.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:02.122+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:02.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:02.122+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:02.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:02.122+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:02.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:02.332+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:02.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:02.332+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:02.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:02.332+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:02.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:02.332+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:02.685+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:02.685+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:02.685+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:02.686+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:03.521 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:03.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:03.521 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:03.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:03.521 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:03.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:03.521 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:03.141+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:03.521 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:03.311+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:03.521 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:03.311+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:03.522 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:03.311+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:03.522 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:03.311+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:03.647+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:03.647+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:03.647+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:03.648+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:04.188+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:04.188+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:04.188+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:04.189+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:04.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:04.286+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:04.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:04.286+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:04.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:04.286+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:04.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:04.287+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:04.605+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:05.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:04.605+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:05.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:04.605+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:05.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:04.606+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:05.433 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:05.321+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:05.434 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:05.321+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:05.434 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:05.321+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:05.434 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:05.322+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:05.434 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:05.142+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:05.434 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:05.142+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:05.434 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:05.142+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:05.434 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:05.143+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:05.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:19:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:19:05.432+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 599 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:19:05.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:05.558+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:05.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:05.558+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:05.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:05.558+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:05.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:05.559+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:06.521 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:06.175+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:06.521 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:06.175+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:06.521 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:06.175+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:06.521 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:06.176+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:06.521 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:06.289+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:06.522 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:06.289+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:06.522 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:06.289+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:06.522 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:06.290+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:06.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:06.519+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:06.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:06.519+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:06.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:06.519+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:06.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:06.520+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:07.549 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:07.246+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:07.549 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:07.246+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:07.549 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:07.246+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:07.549 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:07.246+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:07.549 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:07.165+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:07.549 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:07.165+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:07.549 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:07.165+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:07.549 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:07.166+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:07.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:07.547+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:07.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:07.547+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:07.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:07.547+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:07.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:07.548+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:08.518 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:08.288+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:08.519 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:08.288+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:08.519 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:08.288+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:08.519 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:08.289+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:08.519 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:08.118+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:08.519 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:08.118+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:08.519 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:08.118+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:08.520 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:08.118+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:08.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:08.517+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:08.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:08.517+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:08.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:08.517+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:08.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:08.518+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:09.554 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:09.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:09.554 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:09.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:09.554 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:09.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:09.554 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:09.141+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:09.554 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:09.315+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:09.554 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:09.315+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:09.555 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:09.315+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:09.555 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:09.316+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:09.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:09.552+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:09.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:09.552+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:09.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:09.552+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:09.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:09.553+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:10.434 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:10.296+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:10.434 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:10.296+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:10.434 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:10.296+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:10.434 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:10.297+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:10.434 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:10.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:10.435 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:10.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:10.435 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:10.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:10.435 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:10.137+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:10.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:19:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:19:10.432+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 626 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:19:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:10.590+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:10.590+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:10.590+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:10.591+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:11.307+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:11.307+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:11.307+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:11.308+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:11.153+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:11.153+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:11.153+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:11.154+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:11.596+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:11.596+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:11.596+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:11.597+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:12.333+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:12.333+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:12.333+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:12.333+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:12.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:12.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:12.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:12.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:12.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:12.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:12.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:12.142+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:12.618+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:12.618+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:12.618+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:12.619+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:13.363 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:13.097+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:13.363 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:13.097+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:13.364 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:13.097+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:13.364 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:13.097+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:13.633 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:13.362+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:13.633 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:13.362+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:13.633 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:13.362+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:13.633 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:13.363+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:14.083 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:13.631+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:14.083 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:13.631+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:14.083 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:13.631+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:14.083 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:13.632+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:14.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:14.082+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:14.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:14.082+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:14.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:14.082+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:14.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:14.082+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:14.666 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:14.364+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:14.666 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:14.364+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:14.666 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:14.364+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:14.666 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:14.365+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:14.664+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:14.664+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:14.664+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:14.665+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:15.414 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:15.102+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:15.414 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:15.102+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:15.414 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:15.102+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:15.414 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:15.103+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:15.673 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:19:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:19:15.433+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 655 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:19:15.673 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:15.412+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:15.673 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:15.412+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:15.673 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:15.412+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:15.674 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:15.413+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:15.672+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:15.672+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:15.672+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:15.672+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:16.373+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:16.373+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:16.373+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:16.374+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:16.145+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:16.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:16.145+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:16.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:16.145+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:16.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:16.146+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:16.680+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:16.680+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:16.680+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:16.681+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:17.351+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:17.351+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:17.351+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:17.352+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:17.142+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:17.142+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:17.142+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:17.143+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:17.647+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:17.647+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:17.647+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:17.648+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:18.309+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:18.309+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:18.309+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:18.309+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:18.161+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:18.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:18.161+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:18.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:18.161+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:18.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:18.161+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:18.605+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:18.605+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:18.605+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:18.606+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:19.572 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:19.349+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:19.572 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:19.349+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:19.572 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:19.349+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:19.572 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:19.349+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:19.572 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:19.210+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:19.572 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:19.210+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:19.572 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:19.210+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:19.572 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:19.211+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:19.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:19.570+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:19.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:19.570+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:19.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:19.570+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:19.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:19.571+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:20.558 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:19:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:19:20.433+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 682 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:19:20.558 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:20.357+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:20.558 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:20.357+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:20.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:20.357+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:20.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:20.358+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:20.559 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:20.245+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:20.559 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:20.245+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:20.559 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:20.245+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:20.559 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:20.246+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:20.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:20.612+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:20.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:20.612+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:20.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:20.612+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:20.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:20.612+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:21.386+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:21.386+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:21.386+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:21.387+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:21.205+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:21.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:21.206+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:21.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:21.206+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:21.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:21.206+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:21.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:21.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:21.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:21.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:21.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:21.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:21.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:21.606+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:22.345+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:22.345+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:22.345+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:22.346+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:22.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:22.249+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:22.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:22.249+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:22.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:22.249+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:22.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:22.250+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:22.656+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:22.656+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:22.656+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:22.657+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:23.522 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:23.318+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:23.523 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:23.318+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:23.523 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:23.318+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:23.523 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:23.318+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:23.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:23.208+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:23.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:23.208+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:23.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:23.208+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:23.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:23.209+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:23.647+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:23.647+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:23.647+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:23.648+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:24.280+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:24.280+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:24.280+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:24.280+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:24.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:24.256+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:24.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:24.256+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:24.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:24.256+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:24.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:24.257+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:24.964 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:24.614+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:24.964 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:24.614+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:24.965 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:24.614+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:24.965 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:24.614+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:25.558 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:19:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:19:25.434+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 713 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:19:25.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:25.258+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:25.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:25.258+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:25.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:25.258+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:25.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:25.259+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:25.559 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:25.233+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:25.559 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:25.233+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:25.559 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:25.233+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:25.559 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:25.233+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:25.662+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:25.662+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:25.662+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:25.663+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:26.231+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:26.231+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:26.231+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:26.231+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:26.185+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:26.185+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:26.185+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:26.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:26.186+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:27.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:26.678+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:27.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:26.678+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:27.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:26.678+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:27.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:26.680+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:27.239+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:27.239+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:27.239+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:27.240+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:27.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:27.174+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:27.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:27.174+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:27.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:27.174+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:27.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:27.174+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:28.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:27.645+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:28.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:27.645+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:28.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:27.645+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:28.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:27.646+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:28.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:28.144+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:28.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:28.144+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:28.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:28.144+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:28.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:28.145+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:28.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:28.243+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:28.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:28.243+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:28.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:28.243+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:28.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:28.244+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:28.601+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:28.601+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:28.601+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:28.602+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:29.229+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:29.229+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:29.229+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:29.230+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:29.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:29.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:29.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:29.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:29.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:29.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:29.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:29.133+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:29.899 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:29.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:29.899 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:29.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:29.899 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:29.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:29.899 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:29.628+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:30.436 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:30.199+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:30.436 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:30.199+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:30.436 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:30.199+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:30.436 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:30.199+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:30.436 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:30.160+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:30.436 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:30.160+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:30.437 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:30.160+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:30.437 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:30.161+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:30.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:19:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:19:30.434+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 740 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:19:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:30.607+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:30.607+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:30.607+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:30.608+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:31.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:31.154+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:31.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:31.154+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:31.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:31.154+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:31.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:31.155+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:31.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:31.170+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:31.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:31.170+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:31.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:31.170+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:31.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:31.171+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:31.648+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:31.648+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:31.648+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:31.648+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:32.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:32.121+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:32.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:32.121+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:32.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:32.121+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:32.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:32.122+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:32.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:32.111+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:32.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:32.111+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:32.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:32.111+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:32.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:32.112+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:33.076 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:32.668+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:33.076 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:32.668+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:33.076 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:32.668+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:33.076 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:32.669+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:33.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:33.074+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:33.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:33.074+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:33.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:33.074+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:33.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:33.075+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:33.100+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:33.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:33.100+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:33.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:33.100+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:33.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:33.101+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:33.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:33.647+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:33.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:33.647+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:33.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:33.647+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:33.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:33.647+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:34.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:34.058+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:34.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:34.058+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:34.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:34.058+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:34.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:34.058+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:34.131+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:34.131+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:34.131+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:34.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:34.131+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:35.042 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:34.685+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:35.043 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:34.685+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:35.043 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:34.685+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:35.043 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:34.686+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:35.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:35.041+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:35.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:35.041+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:35.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:35.041+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:35.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:35.041+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:35.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:35.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:35.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:35.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:35.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:35.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:35.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:35.136+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:35.694 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:19:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:19:35.435+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 784 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:19:36.009 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:35.693+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:36.009 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:35.693+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:36.009 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:35.693+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:36.009 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:35.694+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:36.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:36.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:36.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:36.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:36.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:36.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:36.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:36.137+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:36.008+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:36.008+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:36.008+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:36.008+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:36.989 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:36.720+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:36.989 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:36.720+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:36.990 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:36.720+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:36.990 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:36.721+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:37.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:37.163+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:37.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:37.163+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:37.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:37.163+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:37.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:37.164+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:36.988+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:36.988+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:36.988+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:36.989+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:38.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:37.757+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:38.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:37.757+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:38.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:37.757+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:38.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:37.758+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:38.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:37.983+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:38.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:37.983+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:38.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:37.983+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:38.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:37.984+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:38.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:38.213+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:38.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:38.213+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:38.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:38.213+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:38.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:38.214+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:38.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:38.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:38.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:38.795+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:39.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:38.982+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:39.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:38.982+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:39.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:38.982+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:39.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:38.983+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:39.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:39.256+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:39.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:39.256+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:39.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:39.256+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:39.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:39.257+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:40.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:39.983+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:40.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:39.983+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:40.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:39.983+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:40.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:39.983+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:39.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:39.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:40.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:39.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:40.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:39.792+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:40.560 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:19:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:19:40.436+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 811 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:19:40.560 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:40.271+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:40.560 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:40.271+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:40.560 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:40.271+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:40.560 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:40.272+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:41.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:40.992+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:41.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:40.992+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:41.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:40.992+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:41.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:40.993+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:41.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:40.815+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:41.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:40.815+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:41.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:40.815+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:41.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:40.816+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:41.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:41.286+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:41.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:41.286+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:41.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:41.286+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:41.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:41.286+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:41.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:41.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:41.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:41.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:41.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:41.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:41.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:41.837+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:42.270 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:41.948+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:42.270 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:41.948+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:42.270 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:41.948+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:42.270 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:41.949+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:42.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:42.269+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:42.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:42.269+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:42.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:42.269+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:42.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:42.269+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:43.229 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:42.916+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:43.229 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:42.916+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:43.229 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:42.916+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:43.229 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:42.917+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:43.229 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:42.849+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:43.230 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:42.849+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:43.230 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:42.849+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:43.230 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:42.850+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:43.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:43.227+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:43.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:43.227+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:43.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:43.227+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:43.524 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:43.229+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:44.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:43.929+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:44.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:43.929+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:44.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:43.929+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:44.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:43.929+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:44.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:43.829+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:44.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:43.829+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:44.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:43.829+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:44.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:43.830+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:44.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:44.210+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:44.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:44.210+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:44.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:44.210+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:44.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:44.211+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:45.173 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:44.878+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:45.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:44.878+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:45.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:44.878+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:45.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:44.879+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:45.174 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:44.939+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:45.174 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:44.940+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:45.174 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:44.940+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:45.174 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:44.941+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:45.438 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:45.172+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:45.438 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:45.172+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:45.438 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:45.172+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:45.438 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:45.173+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:45.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:19:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:19:45.436+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 840 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:19:46.163 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:45.849+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:46.164 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:45.849+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:46.164 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:45.849+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:46.164 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:45.850+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:46.164 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:45.968+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:46.164 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:45.968+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:46.164 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:45.968+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:46.164 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:45.968+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:46.468 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:46.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:46.468 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:46.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:46.468 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:46.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:46.468 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:46.163+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:47.186 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:46.970+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:47.186 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:46.970+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:47.186 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:46.970+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:47.186 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:46.971+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:47.186 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:46.850+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:47.186 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:46.850+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:47.186 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:46.850+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:47.187 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:46.852+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:47.184+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:47.184+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:47.184+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:47.185+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:48.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:48.018+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:48.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:48.018+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:48.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:48.018+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:48.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:48.018+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:48.170 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:47.888+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:48.170 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:47.888+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:48.170 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:47.888+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:48.171 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:47.889+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:48.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:48.168+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:48.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:48.168+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:48.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:48.168+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:48.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:48.169+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:49.205 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:49.017+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:49.205 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:49.017+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:49.205 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:49.017+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:49.206 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:49.018+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:49.206 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:48.907+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:49.206 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:48.907+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:49.206 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:48.907+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:49.206 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:48.908+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:49.204+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:49.204+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:49.204+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:49.205+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:50.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:50.041+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:50.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:50.041+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:50.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:50.041+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:50.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:50.042+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:50.219 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:49.909+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:50.219 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:49.909+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:50.219 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:49.909+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:50.219 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:49.910+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:50.561 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:19:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:19:50.437+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 867 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:19:50.561 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:50.217+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:50.561 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:50.217+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:50.561 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:50.217+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:50.561 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:50.217+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:51.205 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:50.995+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:51.205 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:50.995+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:51.206 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:50.995+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:51.206 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:50.995+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:51.206 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:50.882+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:51.206 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:50.882+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:51.206 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:50.882+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:51.206 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:50.883+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:51.204+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:51.204+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:51.204+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:51.205+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:52.006 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:51.857+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:52.006 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:51.857+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:52.007 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:51.857+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:52.007 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:51.858+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:52.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:52.249+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:52.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:52.249+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:52.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:52.249+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:52.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:52.250+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:52.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:52.005+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:52.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:52.005+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:52.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:52.005+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:52.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:52.006+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:52.977 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:52.843+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:52.978 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:52.843+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:52.978 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:52.843+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:52.978 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:52.844+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:53.270 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:52.976+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:53.270 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:52.976+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:53.270 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:52.976+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:53.270 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:52.977+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:53.524 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:53.269+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:53.524 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:53.269+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:53.524 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:53.269+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:53.524 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:53.269+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:54.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:53.993+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:54.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:53.993+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:54.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:53.993+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:54.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:53.993+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:54.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:53.821+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:54.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:53.821+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:54.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:53.821+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:54.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:53.822+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:54.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:54.263+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:54.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:54.263+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:54.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:54.263+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:54.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:54.263+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:55.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:54.783+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:55.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:54.783+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:55.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:54.783+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:55.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:54.784+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:55.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:54.998+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:55.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:54.999+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:55.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:54.999+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:55.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:54.999+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:55.561 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:19:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:19:55.439+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 896 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:19:55.562 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:55.243+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:55.562 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:55.243+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:55.562 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:55.243+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:55.562 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:55.243+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:55.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:55.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:55.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:55.792+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:56.203 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:56.012+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:56.203 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:56.012+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:56.204 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:56.012+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:56.204 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:56.012+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:56.203+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:56.203+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:56.203+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:56.204+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:57.039 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:56.771+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:57.039 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:56.771+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:57.039 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:56.771+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:57.039 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:56.772+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:57.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:57.038+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:57.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:57.038+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:57.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:57.038+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:57.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:57.039+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:57.156+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:57.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:57.156+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:57.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:57.156+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:57.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:57.156+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:58.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:57.819+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:58.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:57.819+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:58.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:57.819+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:58.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:57.820+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:58.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:58.054+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:58.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:58.054+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:58.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:58.054+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:58.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:58.054+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:58.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:58.107+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:58.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:58.107+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:58.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:58.107+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:58.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:58.108+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:59.050 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:58.796+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:59.050 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:58.796+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:59.050 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:58.796+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:19:59.050 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:58.797+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:19:59.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:59.049+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:59.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:59.049+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:59.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:59.049+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:19:59.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:19:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:19:59.051+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:19:59.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:59.064+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:59.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:59.064+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:59.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:59.064+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:19:59.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:19:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:19:59.065+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:00.081 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:59.774+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:00.081 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:59.774+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:00.081 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:59.774+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:00.081 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:19:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:19:59.775+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:00.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:00.080+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:00.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:00.080+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:00.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:00.080+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:00.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:00.081+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:00.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:00.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:00.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:00.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:00.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:00.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:00.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:00.107+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:00.788 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:20:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:20:00.439+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 923 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:20:01.064 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:00.788+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:01.064 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:00.788+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:01.064 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:00.788+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:01.064 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:00.789+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:01.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:01.069+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:01.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:01.069+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:01.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:01.069+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:01.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:01.069+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:01.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:01.063+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:01.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:01.063+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:01.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:01.063+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:01.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:01.065+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:02.052 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:01.796+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:02.052 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:01.796+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:02.052 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:01.796+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:02.053 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:01.797+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:02.051+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:02.051+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:02.051+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:02.052+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:02.056+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:02.056+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:02.056+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:02.057+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:03.056 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:02.783+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:03.056 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:02.783+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:03.057 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:02.783+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:03.057 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:02.783+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:03.065+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:03.065+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:03.065+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:03.066+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:03.055+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:03.055+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:03.055+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:03.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:03.056+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:03.766+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:03.766+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:03.766+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:03.767+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:04.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:04.085+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:04.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:04.085+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:04.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:04.085+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:04.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:04.086+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:04.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:04.053+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:04.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:04.053+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:04.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:04.053+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:04.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:04.053+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:05.044 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:04.772+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:05.044 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:04.772+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:05.044 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:04.772+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:05.044 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:04.773+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:05.117+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:05.117+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:05.117+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:05.118+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:05.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:05.043+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:05.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:05.043+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:05.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:05.043+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:05.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:05.043+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:05.795 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:20:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:20:05.439+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 952 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:20:06.080 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:05.793+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:06.081 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:05.793+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:06.081 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:05.793+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:06.081 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:05.794+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:06.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:06.108+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:06.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:06.108+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:06.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:06.108+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:06.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:06.108+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:06.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:06.079+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:06.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:06.079+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:06.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:06.079+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:06.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:06.080+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:07.034 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:06.749+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:07.034 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:06.749+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:07.034 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:06.749+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:07.034 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:06.750+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:07.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:07.061+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:07.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:07.061+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:07.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:07.061+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:07.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:07.062+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:07.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:07.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:07.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:07.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:07.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:07.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:07.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:07.033+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:08.031 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:07.760+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:08.032 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:07.760+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:08.032 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:07.760+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:08.032 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:07.761+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:08.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:08.031+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:08.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:08.031+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:08.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:08.031+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:08.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:08.032+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:08.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:08.030+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:08.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:08.030+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:08.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:08.030+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:08.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:08.031+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:09.037 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:08.770+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:09.037 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:08.770+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:09.037 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:08.770+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:09.037 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:08.771+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:09.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:09.042+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:09.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:09.042+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:09.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:09.042+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:09.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:09.042+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:09.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:09.035+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:09.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:09.035+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:09.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:09.035+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:09.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:09.036+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:10.039 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:09.739+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:10.039 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:09.739+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:10.039 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:09.739+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:10.039 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:09.740+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:10.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:10.037+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:10.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:10.037+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:10.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:10.037+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:10.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:10.038+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:10.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:10.078+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:10.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:10.078+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:10.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:10.078+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:10.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:10.079+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:10.767 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:20:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:20:10.439+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 979 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:20:11.046 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:10.765+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:11.046 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:10.765+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:11.046 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:10.765+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:11.046 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:10.767+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:11.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:11.054+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:11.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:11.054+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:11.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:11.054+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:11.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:11.056+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:11.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:11.098+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:11.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:11.098+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:11.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:11.098+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:11.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:11.098+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:12.038 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:11.752+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:12.039 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:11.752+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:12.039 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:11.752+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:12.039 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:11.753+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:12.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:12.037+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:12.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:12.037+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:12.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:12.037+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:12.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:12.038+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:12.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:12.146+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:12.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:12.146+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:12.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:12.146+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:12.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:12.146+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:13.004+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:13.004+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:13.004+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:13.005+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:12.803+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:12.803+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:12.803+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:12.803+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:13.526 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:13.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:13.526 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:13.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:13.526 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:13.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:13.526 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:13.136+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:14.056 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:13.793+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:14.056 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:13.793+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:14.056 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:13.793+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:14.056 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:13.794+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:14.054+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:14.054+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:14.054+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:14.055+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:14.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:14.107+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:14.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:14.107+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:14.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:14.107+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:14.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:14.108+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:15.013 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:14.830+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:15.013 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:14.830+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:15.013 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:14.830+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:15.013 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:14.830+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:15.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:15.011+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:15.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:15.011+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:15.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:15.011+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:15.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:15.012+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:15.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:15.124+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:15.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:15.124+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:15.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:15.124+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:15.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:15.125+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:15.808 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:20:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:20:15.440+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1008 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:20:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:16.007+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:16.007+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:16.007+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:16.008+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:15.806+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:15.806+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:15.806+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:15.807+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:16.120+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:16.120+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:16.120+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:16.120+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:17.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:17.014+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:17.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:17.014+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:17.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:17.014+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:17.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:17.014+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:16.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:16.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:17.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:16.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:17.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:16.837+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:17.140+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:17.140+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:17.140+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:17.141+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:17.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:17.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:17.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:17.828+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:18.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:17.988+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:18.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:17.988+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:18.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:17.988+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:18.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:17.989+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:18.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:18.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:18.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:18.133+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:19.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:19.005+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:19.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:19.005+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:19.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:19.005+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:19.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:19.006+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:19.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:18.825+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:19.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:18.825+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:19.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:18.825+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:19.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:18.825+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:19.118+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:19.118+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:19.118+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:19.119+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:20.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:19.999+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:20.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:19.999+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:20.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:19.999+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:20.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:19.999+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:20.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:19.789+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:20.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:19.789+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:20.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:19.789+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:20.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:19.789+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:20.442 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:20.110+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:20.442 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:20.110+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:20.442 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:20.110+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:20.442 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:20.111+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:20.789 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:20:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:20:20.440+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1044 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:20:21.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:21.002+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:21.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:21.002+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:21.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:21.002+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:21.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:21.003+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:21.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:20.788+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:21.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:20.788+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:21.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:20.788+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:21.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:20.789+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:21.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:20:21.295+0000 7f4e5afa7700 0 monclient(hunting): authenticate timed out after 300 2024-03-19T03:20:21.296 INFO:teuthology.orchestra.run.smithi059.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-19T03:20:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:21.096+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:21.096+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:21.096+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:21.097+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:21.756 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:21.754+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:21.756 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:21.754+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:21.757 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:21.754+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:21.757 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:21.755+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:21.992+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:21.992+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:21.992+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:22.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:21.993+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:22.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:22.109+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:22.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:22.109+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:22.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:22.109+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:22.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:22.110+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:23.004 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:22.746+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:23.004 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:22.746+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:23.004 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:22.746+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:23.004 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:22.747+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:23.333 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:23.003+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:23.333 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:23.003+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:23.333 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:23.003+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:23.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:23.003+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:23.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:23.096+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:23.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:23.096+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:23.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:23.096+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:23.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:23.097+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:24.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:23.989+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:24.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:23.989+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:24.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:23.989+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:24.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:23.990+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:24.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:23.741+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:24.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:23.741+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:24.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:23.741+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:24.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:23.742+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:24.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:24.133+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:24.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:24.133+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:24.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:24.133+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:24.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:24.134+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:25.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:24.761+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:25.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:24.761+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:25.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:24.761+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:25.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:24.761+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:25.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:24.945+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:25.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:24.945+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:25.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:24.945+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:25.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:24.946+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:25.442 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:25.098+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:25.442 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:25.098+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:25.443 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:25.098+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:25.443 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:25.099+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:25.811 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:20:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:20:25.441+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1090 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:20:26.074 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:25.925+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:26.074 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:25.925+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:26.074 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:25.925+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:26.074 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:25.927+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:26.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:25.809+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:26.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:25.809+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:26.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:25.809+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:26.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:25.810+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:26.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:26.072+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:26.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:26.072+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:26.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:26.072+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:26.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:26.073+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:27.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:26.953+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:27.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:26.953+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:27.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:26.953+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:27.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:26.953+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:27.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:26.809+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:27.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:26.809+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:27.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:26.809+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:27.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:26.810+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:27.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:27.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:27.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:27.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:27.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:27.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:27.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:27.033+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:28.069 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:27.781+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:28.069 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:27.781+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:28.069 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:27.781+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:28.069 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:27.782+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:28.069 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:27.969+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:28.069 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:27.969+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:28.069 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:27.969+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:28.069 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:27.970+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:28.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:28.067+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:28.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:28.067+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:28.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:28.067+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:28.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:28.068+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:29.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:29.012+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:29.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:29.012+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:29.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:29.012+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:29.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:29.013+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:28.790+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:29.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:28.790+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:29.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:28.790+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:29.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:28.791+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:29.089+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:29.089+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:29.089+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:29.090+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:29.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:29.970+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:29.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:29.970+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:29.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:29.970+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:29.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:29.833+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:29.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:29.833+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:29.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:29.833+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:29.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:29.834+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:29.971+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:30.124+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:30.124+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:30.124+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:30.125+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:30.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:20:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:20:30.442+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1135 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:20:31.139 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:30.991+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:31.139 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:30.991+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:31.139 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:30.991+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:31.139 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:30.991+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:31.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:30.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:31.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:30.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:31.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:30.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:31.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:30.848+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:31.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:31.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:31.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:31.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:31.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:31.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:31.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:31.138+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:32.002 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:31.812+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:32.002 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:31.812+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:32.002 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:31.812+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:32.002 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:31.813+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:32.341 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:32.000+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:32.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:32.000+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:32.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:32.000+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:32.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:32.001+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:32.342 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:32.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:32.342 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:32.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:32.342 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:32.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:32.342 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:32.168+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:33.177 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:33.006+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:33.177 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:33.006+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:33.177 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:33.006+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:33.177 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:33.007+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:33.177 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:32.860+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:33.178 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:32.860+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:33.178 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:32.860+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:33.178 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:32.861+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:33.527 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:33.176+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:33.528 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:33.176+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:33.528 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:33.176+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:33.528 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:33.176+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:34.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:33.823+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:34.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:33.823+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:34.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:33.823+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:34.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:33.824+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:34.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:34.000+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:34.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:34.000+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:34.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:34.000+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:34.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:34.001+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:34.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:34.172+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:34.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:34.172+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:34.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:34.172+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:34.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:34.173+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:35.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:34.834+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:35.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:34.834+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:35.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:34.834+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:35.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:34.835+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:35.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:34.969+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:35.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:34.969+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:35.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:34.969+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:35.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:34.970+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:35.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:35.142+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:35.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:35.142+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:35.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:35.142+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:35.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:35.143+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:35.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:20:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:20:35.442+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1182 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:20:36.157 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:35.997+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:36.157 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:35.997+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:36.158 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:35.997+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:36.158 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:35.998+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:36.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:35.879+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:36.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:35.879+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:36.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:35.879+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:36.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:35.880+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:36.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:36.156+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:36.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:36.156+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:36.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:36.156+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:36.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:36.157+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:37.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:37.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:37.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:37.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:37.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:37.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:37.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:37.141+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:37.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:36.926+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:37.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:36.926+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:37.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:36.926+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:37.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:36.927+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:37.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:36.986+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:37.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:36.986+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:37.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:36.986+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:37.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:36.986+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:38.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:38.008+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:38.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:38.008+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:38.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:38.008+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:38.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:38.009+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:38.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:37.956+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:38.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:37.956+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:38.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:37.956+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:38.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:37.957+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:38.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:38.182+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:38.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:38.182+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:38.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:38.182+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:38.341 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:38.182+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:39.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:39.006+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:39.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:39.006+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:39.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:39.006+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:39.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:39.007+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:39.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:39.006+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:39.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:39.006+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:39.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:39.006+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:39.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:39.007+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:39.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:39.197+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:39.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:39.197+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:39.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:39.197+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:39.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:39.197+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:40.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:40.018+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:40.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:40.018+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:40.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:40.018+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:40.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:40.019+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:40.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:40.010+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:40.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:40.010+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:40.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:40.010+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:40.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:40.011+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:40.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:40.231+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:40.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:40.231+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:40.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:40.231+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:40.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:40.232+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:40.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:20:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:20:40.443+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1225 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:20:41.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:41.066+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:41.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:41.066+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:41.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:41.066+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:41.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:41.067+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:41.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:40.969+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:41.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:40.969+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:41.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:40.969+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:41.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:40.969+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:41.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:41.184+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:41.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:41.184+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:41.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:41.184+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:41.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:41.185+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:42.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:42.052+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:42.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:42.052+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:42.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:42.052+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:42.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:42.053+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:42.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:42.005+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:42.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:42.005+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:42.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:42.005+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:42.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:42.005+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:42.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:42.184+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:42.341 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:42.184+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:42.341 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:42.184+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:42.341 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:42.185+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:43.005+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:43.005+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:43.005+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:43.006+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:43.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:42.964+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:43.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:42.964+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:43.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:42.964+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:43.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:42.965+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:43.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:43.156+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:43.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:43.156+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:43.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:43.156+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:43.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:43.157+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:44.007+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:44.376 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:44.007+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:44.377 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:44.007+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:44.377 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:44.009+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:44.377 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:43.934+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:44.377 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:43.934+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:44.377 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:43.934+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:44.377 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:43.935+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:44.378 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:44.138+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:44.378 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:44.138+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:44.378 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:44.138+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:44.378 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:44.138+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:45.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:44.907+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:45.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:44.907+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:45.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:44.907+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:45.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:44.908+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:45.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:45.042+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:45.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:45.042+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:45.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:45.042+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:45.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:45.043+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:45.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:45.161+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:45.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:45.161+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:45.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:45.161+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:45.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:45.162+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:45.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:20:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:20:45.444+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1283 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:20:46.209 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:45.889+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:46.209 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:45.889+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:46.209 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:45.889+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:46.209 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:45.889+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:46.210 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:46.089+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:46.210 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:46.089+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:46.210 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:46.089+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:46.210 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:46.091+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:46.210 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:46.208+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:46.210 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:46.208+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:46.210 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:46.208+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:46.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:46.209+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:47.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:46.935+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:47.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:46.935+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:47.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:46.935+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:47.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:46.935+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:47.247 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:47.074+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:47.247 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:47.074+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:47.248 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:47.074+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:47.248 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:47.075+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:47.245+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:47.245+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:47.245+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:47.246+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:48.271 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:48.057+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:48.271 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:48.057+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:48.271 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:48.057+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:48.272 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:48.058+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:48.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:47.915+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:48.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:47.915+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:48.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:47.915+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:48.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:47.916+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:48.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:48.270+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:48.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:48.270+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:48.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:48.270+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:48.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:48.271+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:49.237 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:49.072+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:49.237 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:49.072+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:49.237 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:49.072+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:49.237 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:49.073+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:49.238 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:48.963+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:49.238 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:48.963+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:49.238 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:48.963+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:49.238 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:48.964+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:49.236+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:49.236+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:49.236+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:49.237+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:50.240 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:49.919+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:50.240 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:49.919+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:50.240 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:49.919+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:50.240 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:49.920+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:50.240 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:50.039+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:50.240 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:50.039+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:50.241 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:50.039+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:50.241 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:50.040+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:50.569 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:20:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:20:50.444+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1376 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:20:50.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:50.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:50.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:50.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:50.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:50.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:50.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:50.239+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:51.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:50.906+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:51.194 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:50.906+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:51.194 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:50.906+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:51.194 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:50.907+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:51.194 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:51.008+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:51.194 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:51.008+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:51.194 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:51.008+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:51.194 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:51.009+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:51.192+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:51.192+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:51.192+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:51.193+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:52.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:52.052+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:52.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:52.052+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:52.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:52.052+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:52.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:52.053+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:52.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:51.925+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:52.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:51.925+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:52.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:51.925+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:52.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:51.926+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:52.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:52.219+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:52.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:52.219+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:52.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:52.219+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:52.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:52.220+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:53.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:53.098+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:53.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:53.098+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:53.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:53.098+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:53.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:53.099+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:53.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:52.968+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:53.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:52.968+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:53.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:52.968+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:53.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:52.969+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:53.529 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:53.219+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:53.529 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:53.219+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:53.529 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:53.219+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:53.529 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:53.220+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:54.076+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:54.076+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:54.076+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:54.077+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:54.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:53.988+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:54.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:53.988+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:54.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:53.988+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:54.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:53.989+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:54.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:54.237+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:54.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:54.237+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:54.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:54.237+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:54.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:54.238+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:55.233 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:55.084+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:55.233 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:55.084+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:55.233 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:55.084+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:55.233 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:55.084+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:55.233 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:54.942+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:55.233 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:54.942+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:55.233 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:54.942+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:55.233 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:54.943+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:55.571 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:20:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:20:55.445+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1473 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:20:55.571 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:55.231+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:55.571 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:55.231+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:55.571 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:55.231+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:55.571 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:55.232+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:56.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:56.087+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:56.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:56.087+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:56.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:56.087+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:56.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:56.088+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:56.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:55.918+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:56.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:55.918+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:56.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:55.918+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:56.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:55.919+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:56.221+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:56.221+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:56.221+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:56.222+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:57.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:56.880+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:57.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:56.880+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:57.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:56.880+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:57.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:56.881+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:57.138+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:57.138+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:57.138+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:57.139+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:57.213+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:57.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:57.214+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:57.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:57.214+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:57.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:57.214+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:58.136 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:57.842+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:58.137 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:57.842+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:58.137 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:57.842+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:58.137 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:57.843+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:58.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:58.207+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:58.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:58.207+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:58.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:58.207+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:58.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:58.207+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:58.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:58.135+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:58.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:58.135+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:58.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:58.135+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:58.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:58.136+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:59.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:58.799+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:59.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:58.799+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:59.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:58.799+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:20:59.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:58.799+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:20:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:59.158+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:59.158+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:59.158+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:20:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:20:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:20:59.158+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:20:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:59.177+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:59.177+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:59.177+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:20:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:20:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:20:59.177+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:00.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:59.838+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:00.140 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:59.838+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:00.140 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:59.838+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:00.140 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:20:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:20:59.839+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:00.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:00.138+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:00.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:00.138+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:00.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:00.138+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:00.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:00.139+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:00.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:00.208+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:00.448 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:00.208+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:00.448 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:00.208+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:00.448 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:00.209+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:00.836 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:21:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:21:00.446+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1566 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:21:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:00.835+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:00.835+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:00.835+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:00.835+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:01.159+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:01.159+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:01.159+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:01.159+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:01.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:01.259+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:01.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:01.259+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:01.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:01.259+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:01.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:01.259+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:02.283 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:02.113+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:02.283 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:02.113+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:02.283 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:02.113+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:02.283 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:02.113+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:02.284 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:01.875+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:02.284 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:01.876+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:02.284 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:01.876+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:02.284 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:01.877+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:02.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:02.282+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:02.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:02.282+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:02.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:02.282+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:02.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:02.283+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:02.830+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:02.830+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:02.830+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:02.831+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:03.530 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:03.160+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:03.530 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:03.160+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:03.530 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:03.160+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:03.530 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:03.161+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:03.530 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:03.289+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:03.530 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:03.289+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:03.530 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:03.289+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:03.531 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:03.290+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:04.115 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:03.853+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:04.115 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:03.853+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:04.115 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:03.853+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:04.115 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:03.853+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:04.338+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:04.338+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:04.338+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:04.339+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:04.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:04.113+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:04.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:04.113+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:04.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:04.113+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:04.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:04.114+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:04.832+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:04.832+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:04.832+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:04.832+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:05.386 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:05.132+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:05.386 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:05.132+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:05.386 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:05.132+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:05.387 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:05.133+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:05.794 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:21:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:21:05.446+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1662 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:21:05.795 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:05.385+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:05.795 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:05.385+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:05.795 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:05.385+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:05.795 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:05.385+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:05.793+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:05.793+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:05.793+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:05.794+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:06.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:06.098+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:06.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:06.098+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:06.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:06.098+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:06.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:06.099+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:06.790 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:06.423+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:06.790 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:06.423+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:06.790 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:06.423+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:06.790 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:06.424+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:06.789+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:06.789+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:06.789+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:06.789+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:07.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:07.125+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:07.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:07.125+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:07.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:07.125+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:07.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:07.126+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:07.807 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:07.449+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:07.807 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:07.449+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:07.807 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:07.449+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:07.807 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:07.450+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:07.805+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:07.805+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:07.805+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:07.807+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:08.500 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:08.141+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:08.500 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:08.141+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:08.500 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:08.141+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:08.500 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:08.141+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:08.791 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:08.498+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:08.791 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:08.498+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:08.791 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:08.498+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:08.792 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:08.499+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:08.790+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:08.790+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:08.790+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:08.791+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:09.498 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:09.183+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:09.498 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:09.183+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:09.498 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:09.183+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:09.498 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:09.184+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:09.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:09.497+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:09.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:09.497+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:09.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:09.497+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:09.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:09.497+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:10.235 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:09.837+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:10.235 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:09.837+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:10.235 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:09.837+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:10.235 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:09.838+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:10.532 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:21:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:21:10.447+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1754 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:21:10.532 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:10.234+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:10.532 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:10.234+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:10.532 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:10.234+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:10.532 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:10.234+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:10.804 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:10.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:10.804 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:10.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:10.804 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:10.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:10.804 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:10.531+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:11.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:10.803+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:11.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:10.803+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:11.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:10.803+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:11.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:10.803+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:11.550 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:11.239+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:11.550 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:11.239+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:11.550 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:11.239+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:11.550 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:11.240+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:11.834 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:11.549+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:11.835 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:11.549+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:11.835 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:11.549+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:11.835 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:11.550+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:11.834+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:11.834+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:11.834+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:11.835+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:12.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:12.244+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:12.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:12.244+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:12.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:12.244+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:12.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:12.245+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:12.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:12.587+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:12.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:12.587+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:12.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:12.587+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:12.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:12.587+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:13.236 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:12.843+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:13.236 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:12.843+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:13.236 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:12.843+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:13.236 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:12.844+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:13.530 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:13.236+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:13.531 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:13.236+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:13.531 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:13.236+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:13.531 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:13.237+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:13.807 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:13.610+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:13.807 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:13.610+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:13.807 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:13.610+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:13.807 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:13.610+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:14.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:13.807+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:14.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:13.807+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:14.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:13.807+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:14.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:13.808+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:14.247+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:14.247+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:14.247+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:14.248+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:14.840+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:15.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:14.840+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:15.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:14.840+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:15.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:14.841+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:15.111 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:14.597+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:15.111 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:14.597+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:15.111 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:14.597+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:15.111 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:14.598+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:15.557 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:21:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:21:15.449+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1850 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:21:15.557 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:15.224+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:15.557 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:15.224+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:15.557 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:15.224+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:15.558 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:15.224+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:15.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:15.595+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:15.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:15.595+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:15.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:15.595+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:15.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:15.596+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:16.212 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:15.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:16.212 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:15.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:16.212 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:15.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:16.212 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:15.848+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:16.549 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:16.212+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:16.549 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:16.212+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:16.549 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:16.212+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:16.549 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:16.212+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:16.817 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:16.549+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:16.817 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:16.549+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:16.817 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:16.549+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:16.817 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:16.550+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:16.816+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:16.816+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:16.816+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:16.817+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:17.566 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:17.165+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:17.566 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:17.165+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:17.566 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:17.165+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:17.567 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:17.166+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:17.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:17.566+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:17.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:17.566+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:17.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:17.566+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:17.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:17.566+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:17.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:17.798+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:17.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:17.798+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:17.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:17.798+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:17.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:17.799+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:18.537 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:18.139+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:18.537 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:18.140+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:18.537 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:18.140+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:18.537 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:18.140+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:18.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:18.536+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:18.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:18.536+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:18.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:18.536+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:18.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:18.536+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:19.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:18.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:18.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:18.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:18.837+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:19.509 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:19.152+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:19.509 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:19.152+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:19.509 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:19.152+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:19.509 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:19.153+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:19.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:19.508+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:19.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:19.508+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:19.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:19.508+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:19.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:19.509+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:20.164 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:19.855+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:20.165 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:19.855+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:20.165 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:19.855+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:20.165 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:19.856+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:20.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:20.163+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:20.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:20.163+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:20.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:20.163+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:20.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:20.164+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:20.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:21:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:21:20.448+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1942 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:21:20.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:20.477+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:20.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:20.477+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:20.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:20.477+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:20.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:20.477+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:21.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:21.118+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:21.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:21.118+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:21.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:21.118+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:21.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:21.119+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:21.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:20.904+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:21.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:20.904+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:21.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:20.904+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:21.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:20.904+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:21.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:21.478+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:21.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:21.478+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:21.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:21.478+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:21.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:21.478+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:22.098+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:22.098+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:22.098+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:22.099+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:22.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:21.878+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:22.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:21.878+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:22.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:21.878+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:22.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:21.879+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:22.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:22.504+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:22.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:22.504+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:22.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:22.504+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:22.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:22.504+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:22.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:22.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:22.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:22.867+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:23.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:23.077+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:23.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:23.077+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:23.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:23.077+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:23.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:23.079+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:23.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:23.460+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:23.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:23.460+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:23.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:23.460+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:23.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:23.461+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:24.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:24.078+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:24.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:24.078+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:24.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:24.078+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:24.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:24.079+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:24.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:23.837+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:24.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:23.837+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:24.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:23.837+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:24.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:23.838+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:24.772 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:24.419+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:24.772 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:24.419+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:24.772 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:24.419+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:24.773 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:24.419+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:25.036 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:24.843+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:25.036 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:24.843+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:25.036 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:24.843+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:25.036 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:24.844+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:25.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:25.034+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:25.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:25.034+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:25.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:25.034+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:25.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:25.035+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:25.810 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:21:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:21:25.449+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2038 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:21:25.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:25.373+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:25.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:25.373+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:25.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:25.373+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:25.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:25.373+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:26.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:26.042+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:26.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:26.042+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:26.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:26.042+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:26.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:26.043+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:26.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:25.809+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:26.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:25.809+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:26.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:25.809+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:26.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:25.810+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:26.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:26.419+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:26.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:26.419+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:26.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:26.419+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:26.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:26.420+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:27.023+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:27.023+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:27.023+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:27.025+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:27.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:26.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:27.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:26.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:27.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:26.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:27.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:26.847+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:27.833 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:27.419+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:27.833 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:27.419+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:27.833 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:27.419+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:27.833 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:27.420+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:28.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:27.992+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:28.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:27.992+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:28.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:27.992+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:28.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:27.993+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:28.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:27.832+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:28.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:27.832+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:28.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:27.832+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:28.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:27.833+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:28.383+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:28.383+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:28.383+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:28.384+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:29.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:28.999+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:29.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:28.999+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:29.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:28.999+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:29.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:29.000+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:29.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:28.861+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:29.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:28.861+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:29.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:28.861+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:29.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:28.862+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:29.341+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:29.341+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:29.341+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:29.342+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:30.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:29.837+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:30.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:29.837+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:30.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:29.837+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:30.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:29.838+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:30.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:30.026+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:30.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:30.026+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:30.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:30.026+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:30.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:30.027+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:30.814 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:21:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:21:30.450+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2130 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:21:30.814 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:30.380+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:30.814 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:30.380+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:30.815 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:30.380+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:30.815 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:30.381+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:31.071 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:30.813+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:31.071 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:30.813+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:31.071 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:30.813+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:31.071 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:30.814+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:31.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:31.070+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:31.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:31.070+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:31.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:31.070+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:31.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:31.070+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:31.645 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:31.410+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:31.645 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:31.410+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:31.645 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:31.410+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:31.645 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:31.410+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:31.785+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:31.785+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:31.785+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:31.786+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:32.372 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:32.099+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:32.373 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:32.099+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:32.373 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:32.099+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:32.373 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:32.099+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:32.738 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:32.371+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:32.738 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:32.371+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:32.738 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:32.371+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:32.738 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:32.372+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:32.736+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:32.736+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:32.736+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:32.737+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:33.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:33.099+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:33.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:33.099+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:33.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:33.099+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:33.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:33.100+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:33.741 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:33.422+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:33.741 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:33.422+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:33.741 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:33.422+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:33.741 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:33.422+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:34.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:33.740+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:34.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:33.740+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:34.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:33.740+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:34.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:33.741+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:34.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:34.074+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:34.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:34.074+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:34.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:34.074+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:34.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:34.074+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:34.773 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:34.396+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:34.773 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:34.396+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:34.773 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:34.396+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:34.774 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:34.397+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:35.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:34.772+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:35.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:34.772+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:35.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:34.772+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:35.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:34.773+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:35.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:35.119+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:35.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:35.119+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:35.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:35.119+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:35.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:35.119+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:35.781 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:21:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:21:35.450+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2226 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:21:35.781 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:35.418+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:35.781 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:35.418+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:35.781 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:35.418+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:35.781 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:35.419+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:35.779+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:35.779+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:35.779+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:35.780+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:36.422 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:36.136+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:36.422 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:36.136+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:36.422 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:36.136+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:36.422 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:36.137+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:36.822 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:36.420+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:36.822 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:36.420+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:36.823 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:36.420+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:36.823 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:36.421+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:37.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:36.821+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:37.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:36.821+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:37.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:36.821+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:37.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:36.822+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:37.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:37.162+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:37.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:37.162+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:37.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:37.162+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:37.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:37.163+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:37.784 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:37.454+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:37.784 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:37.454+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:37.784 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:37.454+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:37.784 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:37.455+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:38.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:37.782+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:38.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:37.782+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:38.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:37.782+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:38.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:37.783+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:38.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:38.148+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:38.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:38.148+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:38.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:38.148+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:38.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:38.149+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:38.799 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:38.449+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:38.800 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:38.449+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:38.800 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:38.449+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:38.800 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:38.450+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:38.798+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:38.798+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:38.798+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:38.799+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:39.490 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:39.137+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:39.491 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:39.137+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:39.491 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:39.137+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:39.491 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:39.137+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:39.752 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:39.489+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:39.752 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:39.489+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:39.752 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:39.489+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:39.752 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:39.490+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:39.750+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:39.750+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:39.750+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:39.751+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:40.443 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:40.129+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:40.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:40.129+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:40.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:40.129+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:40.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:40.130+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:40.769 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:21:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:21:40.451+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2318 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:21:40.769 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:40.442+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:40.769 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:40.442+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:40.770 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:40.442+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:40.770 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:40.443+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:41.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:40.768+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:41.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:40.768+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:41.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:40.768+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:41.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:40.769+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:41.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:41.127+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:41.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:41.127+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:41.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:41.127+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:41.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:41.128+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:41.729 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:41.465+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:41.729 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:41.465+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:41.729 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:41.465+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:41.729 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:41.466+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:42.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:41.728+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:42.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:41.728+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:42.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:41.728+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:42.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:41.729+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:42.501 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:42.163+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:42.501 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:42.163+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:42.501 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:42.163+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:42.501 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:42.163+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:42.779 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:42.499+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:42.779 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:42.499+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:42.779 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:42.499+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:42.779 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:42.500+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:43.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:42.778+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:43.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:42.778+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:43.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:42.778+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:43.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:42.779+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:43.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:43.153+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:43.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:43.153+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:43.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:43.153+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:43.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:43.153+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:43.779 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:43.473+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:43.779 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:43.473+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:43.779 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:43.473+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:43.779 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:43.474+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:44.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:43.777+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:44.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:43.777+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:44.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:43.777+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:44.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:43.779+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:44.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:44.434+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:44.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:44.434+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:44.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:44.434+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:44.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:44.435+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:44.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:44.200+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:44.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:44.200+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:44.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:44.200+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:44.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:44.201+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:45.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:44.813+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:45.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:44.813+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:45.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:44.813+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:45.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:44.814+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:45.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:45.174+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:45.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:45.174+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:45.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:45.174+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:45.454 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:45.175+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:45.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:21:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:21:45.452+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2414 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:21:45.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:45.457+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:45.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:45.457+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:45.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:45.457+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:45.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:45.458+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:46.091 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:45.837+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:46.091 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:45.837+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:46.091 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:45.837+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:46.091 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:45.837+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:46.593 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:46.181+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:46.593 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:46.181+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:46.593 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:46.181+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:46.593 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:46.182+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:46.594 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:46.420+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:46.594 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:46.420+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:46.594 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:46.420+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:46.594 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:46.420+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:47.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:46.871+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:47.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:46.871+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:47.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:46.871+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:47.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:46.872+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:47.424 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:47.158+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:47.424 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:47.158+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:47.424 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:47.158+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:47.424 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:47.159+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:47.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:47.423+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:47.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:47.423+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:47.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:47.423+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:47.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:47.423+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:48.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:48.110+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:48.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:48.110+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:48.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:48.110+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:48.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:48.111+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:48.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:47.886+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:48.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:47.886+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:48.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:47.886+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:48.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:47.887+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:48.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:48.468+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:48.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:48.468+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:48.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:48.468+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:48.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:48.469+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:49.152 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:48.852+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:49.152 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:48.852+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:49.152 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:48.852+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:49.152 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:48.852+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:49.491 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:49.150+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:49.491 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:49.150+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:49.491 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:49.150+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:49.492 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:49.151+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:49.828 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:49.490+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:49.828 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:49.490+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:49.828 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:49.490+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:49.828 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:49.491+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:49.826+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:49.826+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:49.826+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:49.827+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:50.454 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:50.131+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:50.454 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:50.131+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:50.454 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:50.131+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:50.454 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:50.132+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:50.837 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:21:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:21:50.452+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2506 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:21:50.837 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:50.519+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:50.837 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:50.519+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:50.837 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:50.519+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:50.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:50.519+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:51.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:50.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:51.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:50.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:51.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:50.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:51.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:50.837+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:51.524 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:51.090+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:51.524 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:51.090+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:51.525 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:51.090+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:51.525 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:51.091+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:51.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:51.523+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:51.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:51.523+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:51.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:51.523+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:51.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:51.524+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:52.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:51.854+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:52.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:51.854+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:52.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:51.854+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:52.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:51.855+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:52.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:52.102+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:52.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:52.102+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:52.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:52.102+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:52.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:52.103+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:52.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:52.534+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:52.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:52.534+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:52.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:52.534+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:52.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:52.535+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:53.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:53.128+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:53.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:53.128+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:53.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:53.128+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:53.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:53.128+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:53.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:52.898+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:53.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:52.898+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:53.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:52.898+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:53.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:52.899+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:53.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:53.583+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:53.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:53.583+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:53.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:53.583+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:53.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:53.584+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:54.079+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:54.079+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:54.079+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:54.080+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:54.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:53.924+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:54.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:53.924+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:54.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:53.924+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:54.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:53.925+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:54.585+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:54.585+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:54.585+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:54.586+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:55.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:54.913+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:55.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:54.913+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:55.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:54.913+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:55.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:54.914+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:55.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:55.123+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:55.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:55.123+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:55.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:55.123+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:55.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:55.124+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:55.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:21:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:21:55.453+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2602 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:21:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:55.575+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:55.575+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:55.575+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:55.576+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:56.137 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:55.945+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:56.137 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:55.945+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:56.137 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:55.945+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:56.137 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:55.946+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:56.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:56.136+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:56.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:56.136+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:56.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:56.136+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:56.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:56.137+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:56.980 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:56.594+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:56.980 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:56.594+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:56.980 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:56.594+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:56.980 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:56.594+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:57.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:56.978+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:57.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:56.978+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:57.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:56.978+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:57.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:56.979+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:57.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:57.169+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:57.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:57.169+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:57.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:57.169+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:57.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:57.169+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:57.575+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:57.575+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:57.575+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:57.576+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:58.003+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:58.003+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:58.003+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:58.004+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:58.209+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:58.209+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:58.209+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:58.210+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:58.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:58.570+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:58.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:58.570+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:58.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:58.570+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:58.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:58.571+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:59.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:59.035+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:59.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:59.035+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:59.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:59.035+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:21:59.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:21:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:21:59.036+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:21:59.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:59.177+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:59.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:59.177+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:59.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:59.177+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:21:59.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:21:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:21:59.178+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:21:59.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:59.578+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:59.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:59.578+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:59.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:59.578+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:21:59.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:21:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:21:59.578+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:00.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:00.141+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:00.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:00.141+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:00.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:00.141+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:00.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:00.142+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:00.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:00.076+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:00.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:00.076+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:00.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:00.076+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:00.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:00.077+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:00.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:22:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:22:00.454+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2693 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:22:00.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:00.536+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:00.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:00.536+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:00.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:00.536+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:00.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:00.537+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:01.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:01.131+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:01.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:01.131+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:01.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:01.131+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:01.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:01.132+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:01.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:01.087+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:01.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:01.087+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:01.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:01.087+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:01.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:01.088+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:01.637 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:01.493+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:01.637 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:01.493+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:01.637 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:01.493+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:01.637 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:01.494+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:02.114+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:02.114+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:02.114+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:02.114+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:02.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:02.042+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:02.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:02.042+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:02.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:02.042+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:02.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:02.043+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:02.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:02.522+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:02.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:02.522+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:02.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:02.522+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:02.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:02.523+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:03.048+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:03.048+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:03.048+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:03.048+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:03.152+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:03.152+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:03.152+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:03.153+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:03.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:03.487+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:03.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:03.487+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:03.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:03.487+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:03.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:03.488+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:04.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:04.013+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:04.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:04.013+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:04.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:04.013+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:04.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:04.013+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:04.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:04.108+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:04.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:04.108+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:04.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:04.108+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:04.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:04.109+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:04.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:04.469+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:04.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:04.469+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:04.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:04.469+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:04.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:04.470+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:05.108+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:05.108+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:05.108+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:05.109+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:05.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:04.966+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:05.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:04.966+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:05.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:04.966+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:05.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:04.967+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:05.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:22:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:22:05.455+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2789 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:22:05.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:05.509+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:05.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:05.509+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:05.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:05.509+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:05.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:05.509+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:06.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:06.128+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:06.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:06.128+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:06.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:06.128+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:06.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:06.129+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:06.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:05.935+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:06.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:05.935+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:06.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:05.935+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:06.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:05.936+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:06.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:06.466+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:06.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:06.466+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:06.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:06.466+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:06.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:06.467+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:07.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:07.143+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:07.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:07.143+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:07.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:07.143+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:07.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:07.144+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:07.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:06.896+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:07.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:06.896+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:07.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:06.896+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:07.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:06.897+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:07.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:07.507+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:07.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:07.507+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:07.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:07.507+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:07.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:07.507+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:08.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:07.912+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:08.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:07.912+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:08.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:07.912+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:08.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:07.913+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:08.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:08.173+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:08.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:08.173+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:08.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:08.173+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:08.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:08.174+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:08.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:08.473+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:08.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:08.473+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:08.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:08.473+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:08.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:08.474+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:09.209 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:08.943+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:09.209 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:08.943+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:09.209 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:08.943+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:09.209 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:08.944+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:09.449+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:09.449+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:09.449+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:09.450+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:09.208+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:09.208+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:09.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:09.208+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:09.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:09.209+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:10.248 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:09.951+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:10.248 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:09.951+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:10.248 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:09.951+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:10.248 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:09.952+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:10.577 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:22:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:22:10.455+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2880 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:22:10.577 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:10.443+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:10.577 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:10.443+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:10.577 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:10.443+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:10.577 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:10.444+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:10.577 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:10.247+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:10.577 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:10.247+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:10.578 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:10.247+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:10.578 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:10.248+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:11.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:10.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:11.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:10.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:11.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:10.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:11.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:10.958+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:11.220+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:11.220+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:11.220+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:11.222+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:11.404+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:11.404+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:11.404+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:11.405+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:12.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:11.919+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:12.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:11.919+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:12.214 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:11.919+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:12.214 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:11.920+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:12.212+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:12.212+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:12.212+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:12.213+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:12.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:12.380+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:12.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:12.380+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:12.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:12.380+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:12.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:12.381+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:13.180 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:12.877+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:13.181 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:12.877+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:13.181 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:12.877+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:13.181 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:12.878+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:13.534 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:13.179+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:13.535 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:13.179+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:13.535 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:13.179+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:13.535 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:13.180+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:13.535 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:13.357+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:13.535 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:13.357+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:13.535 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:13.357+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:13.535 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:13.358+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:14.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:13.915+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:14.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:13.915+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:14.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:13.915+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:14.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:13.916+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:14.156+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:14.156+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:14.156+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:14.157+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:14.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:14.375+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:14.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:14.375+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:14.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:14.375+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:14.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:14.376+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:15.186 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:14.909+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:15.186 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:14.909+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:15.186 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:14.909+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:15.186 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:14.910+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:15.458 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:15.184+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:15.458 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:15.184+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:15.458 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:15.184+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:15.458 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:15.186+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:15.458 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:15.350+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:15.458 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:15.350+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:15.458 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:15.350+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:15.458 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:15.350+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:15.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:22:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:22:15.456+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2976 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:22:16.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:15.889+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:16.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:15.889+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:16.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:15.889+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:16.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:15.890+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:16.156+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:16.156+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:16.156+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:16.157+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:16.384+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:16.384+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:16.384+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:16.384+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:17.176 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:16.884+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:17.176 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:16.884+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:17.176 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:16.884+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:17.176 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:16.885+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:17.173+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:17.173+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:17.173+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:17.174+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:17.335+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:17.335+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:17.335+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:17.336+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:18.335 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:18.147+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:18.335 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:18.147+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:18.335 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:18.147+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:18.335 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:18.148+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:18.336 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:17.931+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:18.336 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:17.931+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:18.336 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:17.931+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:18.336 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:17.932+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:18.334+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:18.334+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:18.334+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:18.334+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:19.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:19.147+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:19.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:19.147+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:19.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:19.147+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:19.247 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:19.148+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:19.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:18.943+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:19.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:18.943+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:19.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:18.943+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:19.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:18.944+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:19.287+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:19.287+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:19.287+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:19.288+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:20.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:20.119+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:20.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:20.119+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:20.335 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:20.119+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:20.335 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:20.120+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:20.335 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:19.905+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:20.335 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:19.905+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:20.335 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:19.905+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:20.335 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:19.906+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:20.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:22:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:22:20.457+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3068 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:22:20.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:20.332+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:20.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:20.332+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:20.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:20.332+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:20.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:20.333+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:21.310 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:20.916+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:21.311 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:20.916+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:21.311 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:20.916+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:21.311 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:20.917+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:21.311 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:21.133+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:21.311 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:21.133+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:21.311 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:21.133+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:21.311 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:21.134+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:21.309+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:21.309+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:21.309+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:21.310+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:22.287 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:22.133+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:22.288 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:22.133+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:22.288 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:22.133+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:22.288 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:22.133+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:22.288 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:21.943+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:22.288 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:21.943+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:22.288 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:21.943+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:22.288 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:21.945+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:22.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:22.286+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:22.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:22.286+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:22.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:22.286+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:22.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:22.287+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:23.286 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:22.937+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:23.286 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:22.937+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:23.286 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:22.937+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:23.286 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:22.938+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:23.286 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:23.144+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:23.286 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:23.144+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:23.286 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:23.144+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:23.287 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:23.145+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:23.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:23.284+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:23.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:23.284+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:23.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:23.284+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:23.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:23.285+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:24.298 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:24.155+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:24.298 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:24.155+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:24.298 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:24.155+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:24.298 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:24.156+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:24.299 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:23.951+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:24.299 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:23.951+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:24.299 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:23.951+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:24.299 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:23.952+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:24.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:24.297+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:24.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:24.297+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:24.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:24.297+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:24.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:24.297+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:25.294 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:25.155+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:25.294 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:25.155+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:25.294 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:25.155+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:25.294 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:25.156+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:25.294 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:24.991+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:25.295 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:24.991+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:25.295 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:24.991+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:25.295 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:24.991+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:25.583 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:22:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:22:25.458+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3164 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:22:25.583 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:25.293+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:25.583 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:25.293+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:25.583 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:25.293+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:25.583 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:25.293+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:26.288 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:26.142+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:26.289 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:26.142+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:26.289 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:26.142+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:26.289 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:26.143+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:26.289 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:25.973+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:26.289 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:25.973+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:26.289 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:25.973+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:26.289 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:25.973+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:26.288+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:26.288+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:26.288+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:26.288+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:27.280 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:27.138+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:27.280 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:27.138+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:27.280 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:27.138+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:27.280 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:27.138+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:27.280 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:26.994+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:27.281 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:26.994+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:27.281 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:26.994+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:27.281 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:26.994+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:27.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:27.279+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:27.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:27.279+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:27.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:27.279+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:27.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:27.279+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:28.299 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:28.117+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:28.300 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:28.117+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:28.300 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:28.117+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:28.300 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:28.117+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:28.300 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:28.016+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:28.300 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:28.016+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:28.300 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:28.016+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:28.300 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:28.017+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:28.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:28.299+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:28.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:28.299+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:28.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:28.299+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:28.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:28.300+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:29.277 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:29.149+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:29.277 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:29.149+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:29.277 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:29.149+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:29.277 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:29.150+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:29.278 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:29.001+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:29.278 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:29.001+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:29.278 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:29.001+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:29.278 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:29.002+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:29.277+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:29.277+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:29.277+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:29.278+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:30.259 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:30.187+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:30.260 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:30.187+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:30.260 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:30.187+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:30.260 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:30.187+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:30.260 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:30.006+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:30.260 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:30.006+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:30.260 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:30.006+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:30.260 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:30.007+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:30.585 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:22:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:22:30.459+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3255 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:22:30.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:30.259+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:30.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:30.259+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:30.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:30.259+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:30.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:30.260+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:31.268 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:30.994+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:31.268 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:30.994+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:31.268 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:30.994+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:31.269 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:30.994+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:31.269 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:31.192+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:31.269 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:31.192+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:31.269 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:31.192+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:31.269 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:31.192+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:31.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:31.268+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:31.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:31.268+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:31.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:31.268+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:31.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:31.269+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:32.317 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:32.182+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:32.318 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:32.182+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:32.318 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:32.182+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:32.318 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:32.182+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:32.318 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:32.030+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:32.318 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:32.030+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:32.318 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:32.030+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:32.318 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:32.032+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:32.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:32.317+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:32.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:32.317+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:32.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:32.317+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:32.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:32.318+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:33.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:33.156+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:33.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:33.156+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:33.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:33.156+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:33.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:33.157+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:33.335 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:33.068+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:33.335 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:33.068+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:33.335 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:33.068+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:33.335 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:33.070+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:33.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:33.334+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:33.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:33.334+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:33.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:33.334+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:33.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:33.334+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:34.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:34.101+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:34.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:34.101+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:34.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:34.101+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:34.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:34.102+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:34.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:34.202+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:34.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:34.202+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:34.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:34.202+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:34.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:34.203+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:34.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:34.293+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:34.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:34.293+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:34.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:34.293+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:34.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:34.294+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:35.429 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:35.316+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:35.429 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:35.316+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:35.429 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:35.316+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:35.430 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:35.316+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:35.430 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:35.229+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:35.430 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:35.229+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:35.430 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:35.229+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:35.430 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:35.230+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:35.430 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:35.136+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:35.430 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:35.136+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:35.430 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:35.136+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:35.431 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:35.137+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:35.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:22:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:22:35.459+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3351 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:22:36.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:36.086+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:36.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:36.086+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:36.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:36.086+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:36.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:36.087+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:36.223+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:36.223+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:36.223+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:36.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:36.224+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:36.360+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:36.360+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:36.360+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:36.361+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:37.323 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:37.270+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:37.323 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:37.270+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:37.323 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:37.270+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:37.323 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:37.270+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:37.323 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:37.063+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:37.323 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:37.063+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:37.323 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:37.063+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:37.324 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:37.065+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:37.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:37.321+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:37.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:37.321+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:37.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:37.321+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:37.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:37.322+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:38.287 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:38.263+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:38.287 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:38.263+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:38.287 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:38.263+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:38.288 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:38.264+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:38.288 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:38.020+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:38.288 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:38.020+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:38.288 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:38.020+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:38.288 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:38.021+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:38.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:38.286+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:38.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:38.286+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:38.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:38.286+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:38.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:38.286+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:39.304 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:39.289+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:39.305 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:39.289+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:39.305 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:39.289+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:39.305 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:39.289+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:39.305 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:39.047+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:39.305 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:39.047+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:39.306 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:39.047+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:39.306 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:39.047+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:39.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:39.302+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:39.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:39.302+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:39.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:39.302+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:39.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:39.303+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:40.320 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:40.242+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:40.320 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:40.242+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:40.320 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:40.242+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:40.320 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:40.243+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:40.320 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:40.058+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:40.320 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:40.058+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:40.320 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:40.058+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:40.320 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:40.059+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:40.586 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:22:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:22:40.460+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3442 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:22:40.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:40.318+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:40.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:40.318+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:40.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:40.318+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:40.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:40.319+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:41.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:41.108+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:41.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:41.108+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:41.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:41.108+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:41.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:41.109+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:41.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:41.265+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:41.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:41.265+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:41.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:41.265+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:41.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:41.266+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:41.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:41.311+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:41.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:41.311+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:41.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:41.311+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:41.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:41.312+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:42.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:42.148+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:42.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:42.148+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:42.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:42.148+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:42.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:42.149+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:42.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:42.318+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:42.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:42.318+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:42.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:42.318+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:42.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:42.319+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:42.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:42.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:42.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:42.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:42.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:42.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:42.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:42.292+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:43.537 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:43.135+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:43.537 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:43.135+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:43.537 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:43.135+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:43.537 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:43.136+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:43.538 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:43.319+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:43.538 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:43.319+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:43.538 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:43.319+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:43.538 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:43.320+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:43.538 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:43.284+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:43.538 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:43.284+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:43.538 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:43.284+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:43.538 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:43.285+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:44.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:44.094+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:44.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:44.094+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:44.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:44.094+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:44.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:44.095+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:44.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:44.338+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:44.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:44.338+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:44.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:44.338+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:44.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:44.338+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:44.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:44.312+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:44.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:44.312+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:44.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:44.312+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:44.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:44.312+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:45.462 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:45.373+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:45.463 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:45.373+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:45.463 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:45.373+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:45.463 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:45.374+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:45.463 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:45.132+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:45.463 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:45.132+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:45.463 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:45.132+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:45.463 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:45.133+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:45.463 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:45.272+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:45.464 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:45.272+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:45.464 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:45.272+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:45.464 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:45.273+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:45.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:22:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:22:45.461+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3537 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:22:46.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:46.119+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:46.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:46.119+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:46.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:46.119+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:46.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:46.120+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:46.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:46.249+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:46.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:46.249+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:46.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:46.249+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:46.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:46.249+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:46.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:46.335+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:46.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:46.335+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:46.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:46.335+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:46.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:46.336+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:47.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:47.292+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:47.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:47.292+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:47.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:47.292+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:47.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:47.293+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:47.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:47.088+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:47.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:47.088+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:47.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:47.088+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:47.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:47.089+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:47.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:47.272+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:47.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:47.272+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:47.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:47.272+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:47.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:47.272+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:48.279+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:48.279+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:48.279+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:48.280+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:48.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:48.043+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:48.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:48.043+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:48.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:48.043+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:48.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:48.043+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:48.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:48.249+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:48.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:48.249+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:48.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:48.249+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:48.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:48.249+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:49.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:48.994+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:49.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:48.994+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:49.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:48.994+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:49.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:48.995+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:49.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:49.235+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:49.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:49.235+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:49.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:49.235+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:49.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:49.236+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:49.286+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:49.286+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:49.286+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:49.287+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:50.273 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:50.199+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:50.273 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:50.199+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:50.273 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:50.199+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:50.273 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:50.200+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:50.273 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:49.980+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:50.273 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:49.980+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:50.273 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:49.980+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:50.274 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:49.981+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:50.586 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:22:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:22:50.462+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3629 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:22:50.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:50.271+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:50.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:50.271+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:50.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:50.271+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:50.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:50.272+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:51.217 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:50.962+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:51.217 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:50.962+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:51.217 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:50.962+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:51.217 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:50.963+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:51.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:51.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:51.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:51.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:51.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:51.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:51.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:51.273+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:51.216+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:51.216+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:51.216+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:51.217+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:52.249 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:51.975+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:52.249 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:51.975+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:52.249 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:51.975+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:52.250 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:51.977+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:52.250 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:52.204+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:52.250 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:52.204+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:52.250 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:52.204+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:52.250 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:52.205+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:52.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:52.248+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:52.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:52.248+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:52.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:52.248+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:52.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:52.249+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:53.292 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:53.007+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:53.292 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:53.007+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:53.292 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:53.007+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:53.292 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:53.008+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:53.292 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:53.244+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:53.292 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:53.244+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:53.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:53.244+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:53.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:53.245+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:53.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:53.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:53.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:53.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:53.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:53.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:53.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:53.291+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:54.284 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:53.963+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:54.284 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:53.963+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:54.284 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:53.963+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:54.284 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:53.963+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:54.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:54.323+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:54.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:54.323+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:54.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:54.323+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:54.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:54.323+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:54.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:54.282+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:54.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:54.282+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:54.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:54.282+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:54.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:54.283+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:55.250 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:54.930+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:55.250 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:54.930+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:55.250 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:54.930+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:55.250 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:54.931+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:55.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:22:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:22:55.462+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3724 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:22:55.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:55.288+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:55.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:55.288+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:55.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:55.288+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:55.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:55.288+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:55.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:55.248+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:55.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:55.248+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:55.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:55.248+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:55.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:55.249+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:56.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:55.960+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:56.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:55.960+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:56.273 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:55.960+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:56.273 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:55.961+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:56.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:56.318+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:56.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:56.318+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:56.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:56.318+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:56.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:56.319+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:56.271+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:56.271+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:56.271+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:56.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:56.272+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:57.268 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:56.975+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:57.269 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:56.975+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:57.269 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:56.975+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:57.269 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:56.976+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:57.267+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:57.267+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:57.267+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:57.268+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:57.279+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:57.279+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:57.279+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:57.280+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:58.010+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:58.010+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:58.010+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:58.010+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:58.245+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:58.245+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:58.245+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:58.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:58.246+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:58.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:58.251+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:58.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:58.251+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:58.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:58.251+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:58.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:58.251+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:59.301 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:59.255+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:59.301 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:59.255+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:59.302 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:59.255+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:22:59.302 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:22:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:22:59.256+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:22:59.302 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:59.047+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:59.302 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:59.047+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:59.302 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:59.047+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:22:59.302 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:22:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:22:59.048+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:22:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:59.300+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:59.300+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:59.300+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:22:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:22:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:22:59.301+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:00.258 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:00.068+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:00.258 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:00.068+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:00.258 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:00.068+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:00.258 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:00.069+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:00.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:23:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:23:00.463+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3816 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:23:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:00.278+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:00.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:00.278+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:00.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:00.278+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:00.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:00.278+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:00.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:00.257+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:00.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:00.257+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:00.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:00.257+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:00.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:00.257+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:01.312+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:01.312+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:01.312+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:01.313+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:01.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:01.117+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:01.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:01.117+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:01.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:01.117+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:01.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:01.118+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:01.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:01.241+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:01.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:01.241+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:01.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:01.241+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:01.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:01.242+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:02.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:02.328+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:02.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:02.328+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:02.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:02.328+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:02.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:02.329+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:02.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:02.160+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:02.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:02.160+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:02.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:02.160+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:02.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:02.161+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:02.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:02.213+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:02.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:02.213+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:02.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:02.213+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:02.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:02.214+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:03.302+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:03.302+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:03.302+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:03.302+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:03.206+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:03.206+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:03.206+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:03.207+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:03.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:03.200+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:03.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:03.200+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:03.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:03.200+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:03.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:03.201+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:04.297+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:04.297+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:04.297+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:04.298+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:04.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:04.176+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:04.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:04.176+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:04.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:04.176+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:04.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:04.177+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:04.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:04.215+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:04.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:04.215+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:04.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:04.215+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:04.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:04.216+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:05.465 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:05.341+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:05.466 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:05.341+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:05.466 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:05.342+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:05.466 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:05.343+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:05.466 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:05.178+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:05.466 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:05.178+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:05.466 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:05.178+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:05.466 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:05.179+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:05.467 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:05.222+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:05.467 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:05.222+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:05.467 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:05.222+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:05.467 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:05.223+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:05.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:23:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:23:05.464+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3912 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:23:06.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:06.167+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:06.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:06.167+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:06.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:06.167+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:06.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:06.168+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:06.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:06.184+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:06.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:06.184+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:06.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:06.184+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:06.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:06.185+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:06.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:06.323+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:06.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:06.323+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:06.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:06.323+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:06.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:06.324+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:07.300+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:07.300+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:07.300+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:07.300+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:07.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:07.123+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:07.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:07.123+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:07.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:07.123+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:07.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:07.125+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:07.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:07.208+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:07.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:07.208+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:07.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:07.208+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:07.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:07.208+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:08.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:08.270+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:08.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:08.270+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:08.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:08.270+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:08.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:08.271+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:08.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:08.149+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:08.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:08.149+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:08.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:08.149+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:08.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:08.150+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:08.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:08.188+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:08.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:08.188+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:08.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:08.188+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:08.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:08.189+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:09.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:09.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:09.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:09.142+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:09.262+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:09.262+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:09.262+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:09.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:09.263+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:09.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:09.190+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:09.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:09.190+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:09.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:09.190+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:09.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:09.191+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:10.466 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:10.235+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:10.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:10.235+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:10.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:10.235+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:10.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:10.236+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:10.467 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:10.222+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:10.467 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:10.222+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:10.467 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:10.222+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:10.467 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:10.223+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:10.468 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:10.146+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:10.468 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:10.146+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:10.468 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:10.146+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:10.468 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:10.146+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:10.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:23:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:23:10.465+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4004 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:23:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:11.140+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:11.140+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:11.140+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:11.141+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:11.268+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:11.268+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:11.268+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:11.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:11.269+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:11.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:11.194+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:11.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:11.194+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:11.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:11.194+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:11.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:11.195+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:12.261+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:12.261+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:12.261+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:12.262+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:12.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:12.181+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:12.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:12.181+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:12.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:12.181+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:12.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:12.182+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:12.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:12.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:12.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:12.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:12.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:12.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:12.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:12.107+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:13.539 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:13.276+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:13.539 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:13.276+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:13.539 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:13.276+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:13.539 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:13.277+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:13.540 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:13.217+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:13.540 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:13.217+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:13.540 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:13.217+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:13.540 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:13.217+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:13.540 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:13.133+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:13.540 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:13.133+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:13.541 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:13.133+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:13.541 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:13.133+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:14.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:14.161+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:14.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:14.161+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:14.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:14.161+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:14.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:14.161+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:14.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:14.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:14.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:14.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:14.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:14.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:14.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:14.327+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:14.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:14.200+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:14.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:14.200+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:14.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:14.200+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:14.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:14.201+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:15.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:15.325+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:15.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:15.325+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:15.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:15.325+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:15.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:15.326+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:15.468 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:15.224+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:15.468 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:15.224+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:15.468 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:15.224+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:15.468 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:15.224+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:15.468 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:15.156+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:15.468 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:15.156+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:15.468 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:15.156+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:15.468 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:15.157+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:15.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:23:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:23:15.466+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4101 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:23:16.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:16.231+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:16.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:16.231+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:16.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:16.231+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:16.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:16.231+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:16.355+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:16.355+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:16.355+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:16.356+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:16.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:16.166+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:16.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:16.166+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:16.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:16.166+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:16.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:16.167+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:17.406 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:17.148+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:17.406 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:17.148+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:17.407 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:17.148+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:17.407 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:17.149+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:17.407 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:17.281+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:17.407 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:17.281+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:17.407 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:17.281+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:17.407 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:17.282+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:17.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:17.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:17.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:17.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:17.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:17.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:17.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:17.406+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:18.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:18.384+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:18.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:18.384+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:18.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:18.384+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:18.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:18.384+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:18.437 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:18.286+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:18.437 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:18.286+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:18.437 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:18.286+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:18.437 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:18.288+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:18.438 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:18.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:18.438 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:18.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:18.438 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:18.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:18.438 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:18.137+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:19.384 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:19.325+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:19.384 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:19.325+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:19.384 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:19.325+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:19.384 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:19.326+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:19.384 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:19.110+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:19.384 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:19.110+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:19.385 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:19.110+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:19.385 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:19.110+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:19.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:19.382+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:19.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:19.382+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:19.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:19.382+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:19.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:19.384+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:20.429 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:20.315+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:20.429 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:20.315+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:20.429 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:20.315+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:20.429 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:20.315+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:20.429 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:20.144+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:20.430 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:20.144+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:20.430 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:20.144+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:20.430 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:20.145+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:20.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:23:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:23:20.466+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4194 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:23:20.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:20.428+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:20.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:20.428+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:20.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:20.428+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:20.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:20.428+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:21.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:21.359+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:21.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:21.359+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:21.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:21.359+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:21.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:21.360+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:21.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:21.190+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:21.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:21.190+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:21.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:21.190+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:21.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:21.191+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:21.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:21.447+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:21.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:21.447+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:21.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:21.447+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:21.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:21.448+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:22.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:22.403+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:22.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:22.403+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:22.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:22.403+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:22.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:22.404+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:22.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:22.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:22.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:22.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:22.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:22.455+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:22.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:22.208+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:22.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:22.208+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:22.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:22.208+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:22.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:22.208+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:23.540 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:23.473+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:23.540 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:23.473+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:23.540 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:23.473+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:23.541 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:23.473+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:23.541 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:23.419+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:23.541 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:23.419+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:23.541 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:23.419+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:23.541 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:23.420+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:23.541 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:23.251+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:23.541 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:23.252+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:23.542 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:23.252+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:23.542 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:23.252+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:24.580 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:24.422+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:24.580 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:24.422+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:24.580 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:24.422+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:24.580 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:24.423+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:24.581 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:24.441+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:24.581 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:24.441+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:24.581 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:24.441+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:24.581 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:24.442+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:24.581 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:24.276+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:24.581 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:24.276+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:24.581 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:24.276+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:24.581 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:24.277+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:25.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:23:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:23:25.467+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4292 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:23:25.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:25.402+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:25.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:25.402+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:25.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:25.402+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:25.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:25.403+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:25.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:25.466+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:25.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:25.466+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:25.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:25.466+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:25.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:25.467+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:25.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:25.277+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:25.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:25.277+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:25.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:25.277+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:25.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:25.278+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:26.486+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:26.486+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:26.486+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:26.487+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:26.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:26.406+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:26.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:26.406+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:26.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:26.406+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:26.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:26.407+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:26.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:26.294+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:26.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:26.294+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:26.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:26.294+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:26.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:26.294+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:27.441+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:27.441+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:27.442+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:27.442+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:27.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:27.396+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:27.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:27.396+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:27.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:27.396+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:27.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:27.397+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:27.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:27.292+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:27.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:27.292+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:27.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:27.292+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:27.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:27.293+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:28.490+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:28.490+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:28.490+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:28.490+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:28.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:28.425+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:28.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:28.425+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:28.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:28.425+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:28.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:28.426+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:28.340+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:28.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:28.340+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:28.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:28.340+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:28.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:28.340+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:29.424+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:29.424+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:29.424+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:29.425+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:29.460+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:29.460+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:29.460+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:29.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:29.461+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:29.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:29.389+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:29.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:29.389+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:29.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:29.389+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:29.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:29.390+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:30.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:23:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:23:30.468+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4385 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:23:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:30.466+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:30.466+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:30.466+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:30.467+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:30.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:30.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:30.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:30.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:30.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:30.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:30.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:30.433+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:30.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:30.376+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:30.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:30.376+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:30.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:30.376+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:30.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:30.376+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:31.651 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:31.432+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:31.651 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:31.432+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:31.651 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:31.432+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:31.651 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:31.433+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:31.651 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:31.417+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:31.652 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:31.417+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:31.652 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:31.417+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:31.652 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:31.418+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:31.652 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:31.394+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:31.652 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:31.394+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:31.652 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:31.394+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:31.652 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:31.394+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:32.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:32.399+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:32.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:32.399+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:32.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:32.399+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:32.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:32.400+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:32.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:32.383+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:32.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:32.383+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:32.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:32.383+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:32.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:32.384+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:32.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:32.452+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:32.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:32.452+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:32.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:32.452+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:32.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:32.453+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:33.796 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:33.422+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:33.796 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:33.422+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:33.796 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:33.422+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:33.796 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:33.423+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:33.797 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:33.347+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:33.797 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:33.347+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:33.797 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:33.347+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:33.797 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:33.347+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:33.797 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:33.501+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:33.797 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:33.501+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:33.797 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:33.501+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:33.797 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:33.502+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:34.496+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:34.496+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:34.496+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:34.497+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:34.348+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:34.348+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:34.348+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:34.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:34.349+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:34.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:34.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:34.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:34.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:34.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:34.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:34.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:34.402+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:35.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:23:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:23:35.469+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4488 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:23:35.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:35.511+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:35.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:35.511+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:35.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:35.511+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:35.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:35.512+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:35.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:35.348+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:35.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:35.348+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:35.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:35.348+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:35.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:35.348+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:35.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:35.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:35.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:35.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:35.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:35.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:35.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:35.400+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:36.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:36.377+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:36.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:36.377+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:36.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:36.377+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:36.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:36.378+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:36.533+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:36.533+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:36.533+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:36.534+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:36.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:36.364+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:36.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:36.364+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:36.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:36.364+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:36.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:36.365+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:37.353+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:37.353+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:37.353+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:37.354+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:37.541+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:37.541+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:37.541+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:37.541+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:37.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:37.349+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:37.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:37.349+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:37.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:37.349+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:37.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:37.350+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:38.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:38.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:38.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:38.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:38.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:38.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:38.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:38.341+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:38.544+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:38.544+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:38.544+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:38.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:38.544+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:38.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:38.369+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:38.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:38.369+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:38.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:38.369+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:38.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:38.370+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:39.531 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:39.381+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:39.531 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:39.381+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:39.531 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:39.381+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:39.531 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:39.382+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:39.531 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:39.346+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:39.531 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:39.346+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:39.531 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:39.346+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:39.531 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:39.346+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:39.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:39.529+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:39.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:39.529+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:39.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:39.529+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:39.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:39.530+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:40.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:23:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:23:40.469+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4585 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:23:40.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:40.503+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:40.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:40.503+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:40.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:40.503+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:40.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:40.504+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:40.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:40.313+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:40.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:40.313+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:40.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:40.313+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:40.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:40.314+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:40.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:40.337+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:40.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:40.337+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:40.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:40.337+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:40.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:40.338+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:41.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:41.325+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:41.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:41.325+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:41.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:41.325+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:41.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:41.326+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:41.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:41.476+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:41.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:41.476+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:41.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:41.477+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:41.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:41.478+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:41.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:41.267+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:41.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:41.267+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:41.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:41.267+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:41.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:41.268+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:42.505 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:42.306+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:42.505 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:42.306+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:42.505 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:42.306+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:42.506 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:42.307+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:42.506 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:42.247+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:42.506 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:42.247+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:42.506 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:42.247+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:42.506 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:42.247+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:42.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:42.504+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:42.504+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:42.504+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:42.505+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:43.233+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:43.233+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:43.233+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:43.234+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:43.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:43.460+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:43.460+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:43.460+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:43.461+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:43.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:43.348+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:43.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:43.348+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:43.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:43.348+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:43.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:43.349+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:44.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:44.414+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:44.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:44.414+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:44.591 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:44.414+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:44.591 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:44.414+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:44.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:44.390+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:44.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:44.390+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:44.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:44.390+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:44.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:44.391+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:44.592 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:44.189+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:44.592 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:44.189+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:44.592 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:44.189+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:44.592 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:44.190+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:45.427 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:45.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:45.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:45.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:45.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:45.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:45.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:45.360+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:45.428 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:45.158+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:45.428 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:45.158+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:45.428 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:45.158+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:45.428 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:45.159+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:45.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:23:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:23:45.471+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4680 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:23:45.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:45.426+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:45.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:45.426+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:45.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:45.426+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:45.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:45.428+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:46.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:46.175+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:46.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:46.175+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:46.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:46.175+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:46.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:46.175+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:46.393+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:46.393+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:46.393+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:46.394+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:46.374+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:46.374+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:46.374+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:46.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:46.375+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:47.201+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:47.201+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:47.201+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:47.202+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:47.422+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:47.422+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:47.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:47.422+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:47.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:47.423+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:47.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:47.411+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:47.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:47.411+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:47.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:47.411+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:47.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:47.412+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:48.406 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:48.393+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:48.406 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:48.393+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:48.406 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:48.393+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:48.406 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:48.394+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:48.406 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:48.153+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:48.406 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:48.153+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:48.407 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:48.153+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:48.407 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:48.154+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:48.841 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:48.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:48.841 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:48.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:48.841 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:48.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:48.841 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:48.406+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:49.195+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:49.195+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:49.195+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:49.195+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:49.435+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:49.435+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:49.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:49.435+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:49.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:49.436+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:49.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:49.440+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:49.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:49.440+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:49.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:49.440+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:49.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:49.440+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:50.495 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:23:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:23:50.472+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4777 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:23:50.496 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:50.441+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:50.496 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:50.441+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:50.496 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:50.441+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:50.496 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:50.441+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:50.496 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:50.462+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:50.496 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:50.462+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:50.497 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:50.462+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:50.497 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:50.463+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:50.497 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:50.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:50.497 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:50.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:50.497 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:50.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:50.497 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:50.239+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:51.461 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:51.453+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:51.461 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:51.453+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:51.461 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:51.453+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:51.461 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:51.453+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:51.462 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:51.208+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:51.462 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:51.208+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:51.462 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:51.208+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:51.462 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:51.209+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:51.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:51.461+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:51.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:51.461+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:51.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:51.461+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:51.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:51.462+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:52.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:52.429+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:52.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:52.429+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:52.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:52.429+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:52.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:52.430+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:52.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:52.253+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:52.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:52.253+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:52.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:52.253+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:52.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:52.254+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:52.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:52.441+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:52.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:52.442+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:52.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:52.442+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:52.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:52.442+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:53.542 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:53.473+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:53.542 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:53.473+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:53.542 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:53.473+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:53.542 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:53.474+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:53.543 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:53.457+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:53.543 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:53.457+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:53.543 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:53.457+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:53.543 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:53.458+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:53.543 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:53.257+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:53.543 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:53.257+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:53.543 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:53.257+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:53.543 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:53.258+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:54.494 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:54.463+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:54.495 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:54.463+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:54.495 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:54.463+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:54.495 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:54.464+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:54.495 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:54.232+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:54.495 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:54.232+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:54.495 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:54.232+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:54.495 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:54.233+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:54.493+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:54.493+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:54.493+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:54.494+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:55.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:55.213+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:55.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:55.213+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:55.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:55.213+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:55.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:55.213+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:55.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:23:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:23:55.472+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4888 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:23:55.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:55.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:55.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:55.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:55.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:55.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:55.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:55.480+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:55.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:55.494+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:55.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:55.494+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:55.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:55.494+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:55.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:55.495+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:56.526 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:56.492+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:56.526 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:56.492+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:56.526 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:56.492+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:56.527 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:56.493+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:56.527 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:56.255+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:56.527 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:56.255+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:56.527 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:56.255+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:56.527 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:56.255+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:56.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:56.525+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:56.525+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:56.525+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:56.526+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:57.536 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:57.265+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:57.536 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:57.265+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:57.536 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:57.265+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:57.536 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:57.266+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:57.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:57.564+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:57.564+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:57.564+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:57.565+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:57.535+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:57.535+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:57.535+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:57.536+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:58.517 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:58.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:58.517 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:58.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:58.517 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:58.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:58.517 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:58.240+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:58.516+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:58.516+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:58.516+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:58.517+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:58.612+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:58.612+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:58.612+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:58.613+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:59.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:59.287+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:59.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:59.287+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:59.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:59.287+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:23:59.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:23:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:23:59.288+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:23:59.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:59.569+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:59.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:59.569+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:59.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:59.569+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:23:59.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:23:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:23:59.570+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:23:59.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:59.545+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:59.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:59.545+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:59.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:59.545+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:23:59.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:23:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:23:59.546+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:00.496 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:24:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:24:00.473+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4981 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:24:00.496 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:00.238+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:00.497 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:00.238+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:00.497 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:00.238+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:00.497 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:00.239+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:00.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:00.595+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:00.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:00.595+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:00.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:00.595+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:00.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:00.596+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:00.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:00.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:00.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:00.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:00.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:00.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:00.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:00.568+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:01.284+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:01.284+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:01.284+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:01.285+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:02.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:01.599+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:02.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:01.599+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:02.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:01.599+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:02.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:01.600+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:01.646+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:01.646+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:01.646+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:01.646+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:02.554 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:02.246+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:02.554 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:02.246+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:02.554 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:02.246+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:02.554 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:02.246+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:02.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:02.654+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:02.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:02.654+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:02.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:02.654+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:02.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:02.654+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:02.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:02.552+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:02.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:02.552+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:02.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:02.552+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:02.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:02.553+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:03.543 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:03.231+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:03.543 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:03.231+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:03.543 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:03.231+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:03.543 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:03.232+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:03.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:03.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:03.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:03.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:03.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:03.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:03.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:03.588+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:03.619+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:03.619+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:03.619+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:03.620+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:04.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:04.280+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:04.564 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:04.280+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:04.564 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:04.280+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:04.564 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:04.281+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:04.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:04.562+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:04.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:04.562+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:04.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:04.562+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:04.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:04.563+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:04.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:04.582+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:04.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:04.582+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:04.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:04.582+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:04.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:04.582+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:05.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:24:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:24:05.474+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5078 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:24:05.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:05.327+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:05.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:05.327+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:05.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:05.327+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:05.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:05.327+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:05.590+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:05.590+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:05.590+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:05.591+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:05.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:05.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:05.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:05.597+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:06.650 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:06.567+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:06.650 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:06.567+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:06.650 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:06.567+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:06.650 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:06.567+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:06.650 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:06.377+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:06.650 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:06.377+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:06.650 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:06.377+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:06.650 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:06.378+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:07.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:06.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:07.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:06.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:07.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:06.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:07.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:06.647+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:07.644 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:07.561+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:07.644 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:07.561+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:07.644 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:07.561+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:07.644 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:07.562+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:07.644 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:07.356+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:07.644 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:07.356+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:07.645 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:07.356+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:07.645 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:07.357+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:08.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:07.643+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:08.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:07.643+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:08.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:07.643+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:08.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:07.644+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:08.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:08.406+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:08.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:08.406+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:08.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:08.406+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:08.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:08.407+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:08.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:08.543+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:08.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:08.543+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:08.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:08.543+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:08.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:08.544+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:08.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:08.615+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:08.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:08.615+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:08.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:08.615+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:08.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:08.616+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:09.581 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:09.364+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:09.581 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:09.364+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:09.581 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:09.364+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:09.581 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:09.364+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:09.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:09.580+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:09.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:09.580+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:09.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:09.580+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:09.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:09.581+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:09.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:09.652+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:09.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:09.652+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:09.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:09.652+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:09.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:09.653+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:10.682 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:24:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:24:10.475+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5196 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:24:10.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:10.355+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:10.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:10.355+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:10.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:10.355+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:10.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:10.355+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:10.683 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:10.595+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:10.683 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:10.595+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:10.683 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:10.595+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:10.683 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:10.596+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:11.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:10.680+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:11.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:10.681+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:11.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:10.681+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:11.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:10.682+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:11.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:11.326+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:11.326+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:11.326+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:11.327+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:12.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:11.700+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:12.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:11.700+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:12.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:11.700+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:12.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:11.701+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:11.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:11.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:12.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:11.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:12.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:11.616+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:12.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:12.299+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:12.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:12.299+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:12.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:12.299+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:12.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:12.300+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:12.654+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:12.654+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:12.654+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:12.655+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:12.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:12.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:12.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:12.607+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:13.544 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:13.267+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:13.544 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:13.267+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:13.544 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:13.267+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:13.544 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:13.268+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:13.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:13.700+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:13.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:13.700+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:13.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:13.700+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:13.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:13.701+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:13.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:13.573+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:13.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:13.573+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:13.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:13.573+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:13.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:13.574+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:14.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:14.291+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:14.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:14.291+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:14.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:14.291+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:14.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:14.292+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:14.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:14.671+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:14.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:14.671+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:14.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:14.671+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:14.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:14.672+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:14.562+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:14.562+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:14.562+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:14.563+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:15.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:24:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:24:15.475+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5315 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:24:15.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:15.275+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:15.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:15.275+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:15.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:15.275+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:15.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:15.276+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:15.645+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:15.645+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:15.645+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:15.646+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:15.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:15.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:15.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:15.608+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:16.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:16.320+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:16.320+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:16.320+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:16.321+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:17.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:16.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:17.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:16.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:17.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:16.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:17.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:16.661+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:16.628+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:16.628+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:16.628+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:16.628+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:17.652 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:17.363+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:17.652 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:17.363+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:17.652 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:17.363+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:17.652 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:17.363+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:18.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:17.665+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:18.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:17.665+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:18.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:17.665+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:18.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:17.665+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:17.651+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:17.651+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:17.651+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:18.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:17.652+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:18.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:18.334+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:18.334+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:18.334+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:18.335+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:19.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:18.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:19.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:18.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:19.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:18.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:19.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:18.661+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:18.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:18.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:18.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:18.603+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:19.384+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:19.384+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:19.384+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:19.385+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:19.877 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:19.632+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:19.877 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:19.632+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:19.877 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:19.632+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:19.877 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:19.632+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:19.877 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:19.619+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:19.877 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:19.619+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:19.877 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:19.619+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:19.878 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:19.619+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:20.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:24:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:24:20.476+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5408 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:24:20.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:20.578+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:20.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:20.578+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:20.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:20.578+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:20.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:20.579+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:20.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:20.393+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:20.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:20.393+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:20.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:20.393+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:20.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:20.394+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:20.841 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:20.638+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:20.841 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:20.638+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:20.841 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:20.638+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:20.841 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:20.639+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:21.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:21.593+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:21.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:21.594+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:21.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:21.594+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:21.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:21.595+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:21.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:21.439+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:21.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:21.439+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:21.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:21.439+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:21.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:21.440+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:21.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:21.540+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:21.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:21.540+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:21.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:21.540+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:21.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:21.540+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:22.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:22.567+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:22.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:22.567+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:22.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:22.567+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:22.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:22.568+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:22.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:22.542+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:22.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:22.542+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:22.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:22.542+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:22.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:22.542+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:22.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:22.470+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:22.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:22.470+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:22.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:22.470+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:22.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:22.470+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:23.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:23.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:23.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:23.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:23.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:23.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:23.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:23.589+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:23.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:23.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:23.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:23.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:23.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:23.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:23.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:23.607+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:23.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:23.461+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:23.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:23.461+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:23.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:23.461+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:23.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:23.461+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:24.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:24.576+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:24.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:24.576+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:24.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:24.576+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:24.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:24.577+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:24.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:24.462+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:24.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:24.462+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:24.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:24.462+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:24.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:24.463+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:24.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:24.580+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:24.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:24.580+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:24.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:24.580+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:24.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:24.581+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:25.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:24:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:24:25.477+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5502 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:24:25.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:25.587+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:25.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:25.587+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:25.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:25.587+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:25.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:25.587+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:25.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:25.599+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:25.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:25.599+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:25.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:25.599+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:25.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:25.600+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:25.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:25.439+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:25.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:25.439+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:25.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:25.439+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:25.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:25.441+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:26.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:26.606+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:26.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:26.606+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:26.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:26.606+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:26.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:26.607+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:26.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:26.471+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:26.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:26.471+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:26.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:26.471+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:26.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:26.472+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:26.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:26.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:26.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:26.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:26.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:26.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:26.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:26.628+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:27.509+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:27.509+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:27.509+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:27.510+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:27.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:27.654+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:27.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:27.655+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:27.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:27.655+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:27.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:27.655+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:27.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:27.586+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:27.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:27.586+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:27.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:27.586+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:27.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:27.587+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:28.552+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:28.552+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:28.552+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:28.553+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:28.635+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:28.635+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:28.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:28.635+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:28.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:28.636+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:28.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:28.549+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:28.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:28.549+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:28.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:28.549+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:28.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:28.550+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:29.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:29.669+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:29.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:29.669+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:29.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:29.669+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:29.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:29.670+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:29.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:29.525+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:29.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:29.525+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:29.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:29.525+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:29.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:29.526+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:29.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:29.539+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:29.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:29.540+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:29.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:29.540+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:29.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:29.540+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:30.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:24:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:24:30.478+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5594 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:24:30.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:30.568+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:30.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:30.568+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:30.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:30.568+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:30.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:30.569+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:30.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:30.485+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:30.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:30.485+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:30.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:30.485+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:30.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:30.487+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:30.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:30.698+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:30.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:30.698+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:30.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:30.698+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:30.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:30.699+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:31.729 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:31.459+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:31.729 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:31.459+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:31.729 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:31.459+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:31.730 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:31.460+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:31.730 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:31.607+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:31.730 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:31.607+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:31.730 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:31.607+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:31.730 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:31.608+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:32.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:31.728+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:32.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:31.728+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:32.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:31.728+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:32.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:31.729+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:32.749 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:32.491+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:32.749 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:32.491+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:32.749 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:32.491+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:32.749 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:32.491+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:32.749 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:32.605+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:32.750 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:32.605+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:32.750 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:32.605+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:32.750 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:32.605+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:33.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:32.747+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:33.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:32.747+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:33.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:32.748+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:33.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:32.748+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:33.787 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:33.461+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:33.787 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:33.461+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:33.788 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:33.461+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:33.788 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:33.462+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:33.788 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:33.583+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:33.788 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:33.583+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:33.788 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:33.583+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:33.788 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:33.583+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:34.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:33.786+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:34.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:33.786+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:34.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:33.786+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:34.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:33.787+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:34.763 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:34.447+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:34.763 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:34.447+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:34.763 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:34.447+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:34.763 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:34.448+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:34.764 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:34.625+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:34.764 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:34.625+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:34.764 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:34.625+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:34.764 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:34.626+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:35.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:34.762+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:35.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:34.762+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:35.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:34.762+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:35.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:34.763+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:35.494 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:24:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:24:35.479+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5688 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:24:35.494 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:35.402+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:35.494 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:35.403+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:35.494 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:35.403+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:35.494 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:35.404+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:35.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:35.660+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:35.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:35.660+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:35.812 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:35.660+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:35.812 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:35.660+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:36.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:35.810+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:36.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:35.810+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:36.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:35.810+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:36.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:35.811+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:36.677 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:36.404+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:36.677 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:36.404+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:36.677 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:36.404+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:36.677 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:36.405+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:36.676+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:36.676+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:36.676+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:36.677+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:37.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:36.858+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:37.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:36.858+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:37.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:36.858+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:37.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:36.859+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:37.678 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:37.411+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:37.678 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:37.411+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:37.679 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:37.411+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:37.679 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:37.412+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:38.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:37.864+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:38.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:37.864+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:38.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:37.864+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:38.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:37.864+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:38.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:37.677+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:38.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:37.677+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:38.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:37.677+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:38.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:37.678+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:38.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:38.440+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:38.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:38.440+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:38.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:38.440+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:38.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:38.441+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:38.719+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:38.719+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:38.719+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:38.720+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:39.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:38.854+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:39.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:38.854+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:39.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:38.854+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:39.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:38.855+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:39.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:39.400+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:39.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:39.400+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:39.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:39.400+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:39.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:39.401+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:40.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:39.890+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:40.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:39.890+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:40.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:39.890+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:40.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:39.890+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:39.716+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:39.716+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:39.716+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:39.717+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:40.683 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:24:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:24:40.480+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5780 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:24:40.684 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:40.430+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:40.684 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:40.430+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:40.684 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:40.430+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:40.684 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:40.430+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:41.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:40.682+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:41.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:40.682+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:41.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:40.682+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:41.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:40.683+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:41.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:40.847+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:41.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:40.847+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:41.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:40.847+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:41.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:40.848+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:41.654 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:41.396+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:41.654 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:41.396+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:41.654 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:41.396+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:41.654 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:41.397+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:42.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:41.824+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:42.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:41.824+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:42.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:41.824+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:42.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:41.825+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:42.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:41.669+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:42.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:41.669+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:42.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:41.669+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:42.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:41.670+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:42.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:42.436+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:42.436+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:42.436+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:42.437+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:42.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:42.621+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:42.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:42.621+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:42.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:42.621+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:42.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:42.622+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:42.843+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:42.843+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:42.843+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:42.844+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:43.826 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:43.631+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:43.826 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:43.631+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:43.826 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:43.631+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:43.826 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:43.632+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:43.826 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:43.388+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:43.826 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:43.388+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:43.826 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:43.388+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:43.827 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:43.389+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:44.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:43.824+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:44.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:43.824+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:44.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:43.824+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:44.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:43.825+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:44.680 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:44.429+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:44.680 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:44.429+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:44.680 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:44.429+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:44.680 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:44.430+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:44.809+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:45.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:44.809+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:45.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:44.809+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:45.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:44.810+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:44.679+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:44.679+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:44.679+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:44.680+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:45.661 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:24:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:24:45.481+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5874 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:24:45.661 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:45.403+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:45.661 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:45.403+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:45.661 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:45.403+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:45.661 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:45.404+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:46.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:45.815+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:46.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:45.815+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:46.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:45.815+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:46.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:45.816+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:46.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:45.660+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:46.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:45.660+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:46.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:45.660+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:46.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:45.660+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:46.703 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:46.372+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:46.704 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:46.372+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:46.704 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:46.372+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:46.704 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:46.373+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:47.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:46.788+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:47.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:46.788+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:47.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:46.788+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:47.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:46.788+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:46.702+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:46.702+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:46.702+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:46.703+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:47.336+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:47.336+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:47.336+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:47.336+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:48.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:47.748+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:48.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:47.748+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:48.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:47.748+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:48.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:47.748+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:47.705+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:47.705+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:47.705+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:47.706+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:48.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:48.336+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:48.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:48.336+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:48.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:48.336+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:48.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:48.337+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:49.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:48.734+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:49.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:48.734+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:49.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:48.734+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:49.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:48.735+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:49.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:48.665+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:49.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:48.665+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:49.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:48.665+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:49.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:48.666+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:49.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:49.319+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:49.319+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:49.319+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:49.320+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:49.759+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:49.759+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:49.759+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:49.760+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:50.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:49.683+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:50.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:49.683+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:50.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:49.683+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:50.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:49.684+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:50.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:24:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:24:50.482+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5966 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:24:50.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:50.291+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:50.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:50.291+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:50.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:50.291+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:50.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:50.292+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:50.786+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:50.786+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:50.786+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:50.787+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:50.688+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:50.688+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:50.688+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:50.689+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:51.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:51.306+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:51.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:51.306+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:51.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:51.306+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:51.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:51.306+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:51.682+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:51.682+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:51.682+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:51.683+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:51.761+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:51.761+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:51.761+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:51.762+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:52.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:52.284+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:52.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:52.284+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:52.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:52.284+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:52.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:52.285+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:53.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:52.760+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:53.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:52.760+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:53.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:52.760+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:53.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:52.761+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:52.672+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:52.672+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:52.672+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:52.673+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:53.547 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:53.237+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:53.547 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:53.237+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:53.547 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:53.237+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:53.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:53.238+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:53.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:53.716+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:53.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:53.716+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:53.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:53.716+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:53.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:53.716+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:53.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:53.737+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:53.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:53.737+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:53.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:53.737+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:53.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:53.738+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:54.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:54.277+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:54.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:54.277+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:54.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:54.277+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:54.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:54.278+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:55.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:54.778+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:55.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:54.778+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:55.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:54.778+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:55.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:54.779+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:55.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:54.694+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:55.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:54.694+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:55.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:54.694+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:55.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:54.695+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:55.496 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:24:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:24:55.483+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6060 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:24:55.496 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:55.241+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:55.496 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:55.241+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:55.496 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:55.241+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:55.496 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:55.242+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:55.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:55.740+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:55.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:55.740+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:55.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:55.740+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:55.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:55.740+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:55.741+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:55.741+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:55.741+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:55.742+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:56.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:56.271+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:56.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:56.271+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:56.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:56.271+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:56.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:56.272+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:57.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:56.743+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:57.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:56.743+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:57.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:56.743+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:57.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:56.744+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:57.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:56.786+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:57.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:56.786+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:57.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:56.786+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:57.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:56.787+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:57.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:57.294+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:57.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:57.294+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:57.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:57.294+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:57.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:57.295+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:58.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:57.792+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:58.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:57.792+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:58.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:57.792+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:58.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:57.793+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:57.738+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:57.738+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:57.739+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:57.739+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:58.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:58.321+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:58.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:58.321+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:58.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:58.321+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:58.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:58.322+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:24:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:58.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:58.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:58.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:24:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:58.764+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:59.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:58.800+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:59.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:58.800+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:59.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:58.800+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:24:59.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:58.801+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:24:59.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:59.367+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:59.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:59.367+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:59.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:59.367+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:24:59.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:24:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:24:59.368+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:59.758+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:59.758+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:59.758+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:24:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:24:59.759+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:59.851+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:59.851+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:59.851+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:24:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:24:59.852+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:00.610 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:25:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:25:00.483+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6152 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:25:00.611 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:00.354+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:00.611 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:00.354+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:00.611 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:00.354+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:00.611 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:00.354+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:00.867 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:00.745+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:00.869 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:00.745+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:00.869 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:00.745+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:00.869 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:00.746+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:01.332 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:00.866+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:01.333 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:00.866+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:01.333 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:00.866+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:01.333 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:00.866+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:01.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:01.331+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:01.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:01.331+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:01.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:01.331+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:01.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:01.332+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:02.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:01.748+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:02.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:01.748+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:02.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:01.748+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:02.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:01.748+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:02.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:01.848+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:02.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:01.848+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:02.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:01.848+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:02.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:01.849+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:02.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:02.377+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:02.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:02.377+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:02.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:02.377+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:02.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:02.378+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:03.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:02.894+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:03.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:02.894+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:03.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:02.894+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:03.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:02.895+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:02.720+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:02.720+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:02.720+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:02.721+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:03.685 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:03.364+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:03.685 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:03.364+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:03.685 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:03.364+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:03.685 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:03.364+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:04.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:03.856+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:04.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:03.856+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:04.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:03.856+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:04.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:03.857+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:04.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:03.683+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:04.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:03.683+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:04.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:03.683+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:04.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:03.684+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:04.643 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:04.338+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:04.643 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:04.338+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:04.643 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:04.338+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:04.643 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:04.338+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:04.897 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:04.641+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:04.897 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:04.641+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:04.897 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:04.641+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:04.897 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:04.642+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:05.327 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:04.895+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:05.328 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:04.895+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:05.328 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:04.895+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:05.328 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:04.897+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:05.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:25:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:25:05.485+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6246 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:25:05.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:05.327+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:05.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:05.327+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:05.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:05.327+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:05.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:05.327+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:05.866 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:05.670+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:05.866 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:05.670+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:05.866 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:05.670+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:05.866 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:05.671+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:06.315 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:05.866+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:06.315 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:05.866+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:06.315 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:05.866+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:06.315 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:05.866+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:06.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:06.315+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:06.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:06.315+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:06.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:06.315+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:06.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:06.316+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:07.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:06.908+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:07.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:06.908+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:07.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:06.908+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:07.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:06.909+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:06.695+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:07.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:06.695+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:07.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:06.695+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:07.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:06.695+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:07.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:07.331+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:07.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:07.331+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:07.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:07.331+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:07.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:07.332+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:07.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:07.681+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:07.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:07.681+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:07.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:07.681+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:07.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:07.682+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:08.284 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:07.951+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:08.284 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:07.951+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:08.284 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:07.951+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:08.284 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:07.952+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:08.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:08.284+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:08.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:08.284+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:08.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:08.284+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:08.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:08.284+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:09.030 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:08.911+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:09.355 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:08.911+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:09.355 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:08.911+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:09.355 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:08.912+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:09.356 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:08.680+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:09.356 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:08.680+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:09.356 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:08.680+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:09.356 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:08.681+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:09.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:09.273+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:09.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:09.273+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:09.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:09.273+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:09.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:09.274+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:09.955 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:09.696+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:09.955 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:09.696+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:09.955 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:09.696+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:09.955 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:09.697+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:10.259 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:09.955+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:10.259 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:09.955+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:10.259 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:09.955+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:10.259 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:09.955+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:10.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:25:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:25:10.486+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6338 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:25:10.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:10.258+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:10.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:10.259+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:10.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:10.259+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:10.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:10.260+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:10.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:10.651+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:10.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:10.651+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:10.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:10.651+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:10.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:10.652+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:11.282 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:10.971+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:11.282 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:10.971+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:11.282 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:10.971+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:11.283 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:10.972+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:11.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:11.282+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:11.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:11.282+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:11.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:11.282+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:11.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:11.283+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:11.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:11.617+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:11.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:11.617+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:11.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:11.617+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:11.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:11.617+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:12.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:11.969+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:12.247 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:11.969+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:12.247 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:11.969+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:12.247 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:11.970+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:12.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:12.246+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:12.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:12.246+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:12.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:12.246+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:12.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:12.247+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:12.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:12.594+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:12.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:12.594+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:12.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:12.594+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:12.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:12.594+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:13.227 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:12.950+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:13.227 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:12.950+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:13.227 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:12.950+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:13.227 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:12.951+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:13.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:13.225+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:13.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:13.225+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:13.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:13.225+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:13.549 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:13.226+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:13.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:13.567+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:13.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:13.567+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:13.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:13.567+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:13.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:13.568+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:13.953+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:13.953+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:13.953+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:13.954+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:14.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:14.179+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:14.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:14.179+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:14.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:14.179+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:14.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:14.180+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:14.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:14.585+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:14.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:14.585+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:14.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:14.585+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:14.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:14.586+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:15.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:14.964+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:15.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:14.964+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:15.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:14.964+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:15.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:14.964+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:15.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:15.197+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:15.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:15.197+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:15.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:15.197+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:15.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:15.198+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:15.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:25:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:25:15.486+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6432 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:25:15.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:15.622+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:15.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:15.622+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:15.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:15.622+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:15.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:15.623+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:16.248 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:15.920+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:16.249 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:15.920+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:16.249 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:15.920+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:16.249 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:15.921+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:16.586 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:16.247+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:16.586 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:16.247+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:16.586 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:16.247+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:16.586 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:16.248+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:16.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:16.584+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:16.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:16.584+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:16.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:16.584+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:16.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:16.585+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:17.283 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:16.904+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:17.283 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:16.904+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:17.283 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:16.904+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:17.283 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:16.905+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:17.282+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:17.282+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:17.282+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:17.283+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:17.917 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:17.596+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:17.917 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:17.596+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:17.917 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:17.596+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:17.917 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:17.597+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:18.328 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:17.916+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:18.328 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:17.916+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:18.328 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:17.916+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:18.328 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:17.917+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:18.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:18.326+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:18.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:18.326+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:18.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:18.326+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:18.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:18.327+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:18.915 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:18.628+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:18.916 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:18.628+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:18.916 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:18.628+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:18.916 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:18.629+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:19.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:18.914+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:19.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:18.914+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:19.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:18.914+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:19.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:18.915+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:19.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:19.361+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:19.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:19.361+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:19.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:19.361+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:19.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:19.363+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:19.923 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:19.604+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:19.923 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:19.604+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:19.923 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:19.604+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:19.923 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:19.605+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:19.921+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:19.921+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:19.921+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:19.922+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:20.611 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:25:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:25:20.487+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6524 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:25:20.612 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:20.350+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:20.612 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:20.350+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:20.612 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:20.350+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:20.612 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:20.351+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:20.612 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:20.566+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:20.612 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:20.566+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:20.612 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:20.566+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:20.612 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:20.567+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:21.317 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:20.875+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:21.317 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:20.875+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:21.317 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:20.875+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:21.317 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:20.876+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:21.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:25:21.402+0000 7f64abd21700 0 monclient(hunting): authenticate timed out after 300 2024-03-19T03:25:21.403 INFO:teuthology.orchestra.run.smithi059.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-19T03:25:21.657 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:21.395+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:21.657 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:21.395+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:21.657 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:21.395+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:21.658 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:21.396+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:21.658 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:21.544+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:21.658 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:21.544+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:21.658 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:21.544+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:21.658 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:21.545+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:22.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:21.837+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:22.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:21.837+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:22.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:21.837+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:22.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:21.838+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:22.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:22.385+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:22.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:22.385+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:22.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:22.385+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:22.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:22.385+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:22.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:22.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:22.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:22.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:22.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:22.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:22.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:22.531+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:23.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:22.844+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:23.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:22.844+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:23.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:22.844+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:23.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:22.845+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:23.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:23.483+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:23.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:23.483+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:23.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:23.483+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:23.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:23.483+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:23.350+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:23.350+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:23.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:23.350+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:23.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:23.351+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:24.324 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:23.841+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:24.324 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:23.841+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:24.324 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:23.841+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:24.325 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:23.841+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:24.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:24.517+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:24.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:24.517+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:24.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:24.517+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:24.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:24.519+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:24.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:24.323+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:24.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:24.323+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:24.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:24.323+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:24.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:24.324+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:25.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:24.874+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:25.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:24.874+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:25.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:24.874+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:25.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:24.875+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:25.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:25:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:25:25.488+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6618 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:25:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:25.363+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:25.363+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:25.363+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:25.364+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:25.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:25.526+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:25.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:25.526+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:25.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:25.526+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:25.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:25.527+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:26.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:25.868+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:26.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:25.868+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:26.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:25.868+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:26.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:25.869+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:26.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:26.383+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:26.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:26.383+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:26.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:26.383+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:26.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:26.384+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:26.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:26.540+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:26.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:26.540+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:26.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:26.540+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:26.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:26.541+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:26.906+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:26.906+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:26.906+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:26.906+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:27.574+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:27.575+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:27.575+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:27.575+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:27.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:27.400+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:27.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:27.400+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:27.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:27.400+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:27.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:27.400+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:28.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:27.952+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:27.952+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:27.952+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:27.953+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:28.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:28.398+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:28.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:28.398+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:28.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:28.398+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:28.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:28.398+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:28.602+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:28.602+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:28.602+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:28.603+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:29.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:28.939+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:29.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:28.939+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:29.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:28.939+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:29.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:28.939+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:29.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:29.354+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:29.354+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:29.354+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:29.354+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:29.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:29.601+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:29.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:29.601+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:29.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:29.601+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:29.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:29.601+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:30.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:29.973+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:29.973+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:29.973+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:29.974+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:30.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:25:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:25:30.489+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6710 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:25:30.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:30.568+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:30.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:30.568+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:30.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:30.568+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:30.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:30.569+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:30.395+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:30.395+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:30.395+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:30.396+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:31.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:30.926+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:31.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:30.926+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:31.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:30.926+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:31.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:30.927+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:31.598 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:31.432+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:31.599 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:31.432+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:31.599 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:31.432+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:31.599 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:31.433+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:31.913 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:31.597+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:31.914 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:31.597+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:31.914 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:31.597+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:31.914 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:31.598+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:32.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:31.912+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:32.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:31.912+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:32.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:31.912+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:32.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:31.913+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:32.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:32.552+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:32.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:32.552+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:32.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:32.552+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:32.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:32.552+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:32.387+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:32.387+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:32.387+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:32.388+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:33.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:32.914+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:33.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:32.914+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:33.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:32.914+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:33.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:32.915+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:33.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:33.556+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:33.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:33.556+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:33.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:33.556+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:33.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:33.556+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:33.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:33.406+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:33.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:33.406+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:33.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:33.406+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:33.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:33.407+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:34.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:33.913+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:34.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:33.913+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:34.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:33.913+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:34.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:33.913+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:34.528+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:34.528+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:34.528+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:34.528+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:34.394+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:34.394+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:34.394+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:34.395+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:35.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:34.918+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:35.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:34.918+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:35.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:34.918+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:35.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:34.919+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:35.615 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:25:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:25:35.490+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6804 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:25:35.616 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:35.364+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:35.616 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:35.364+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:35.616 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:35.364+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:35.616 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:35.365+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:35.616 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:35.483+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:35.616 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:35.483+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:35.616 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:35.483+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:35.617 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:35.484+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:36.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:35.968+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:35.968+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:35.968+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:35.969+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:36.357+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:10.129136+0000 front 2024-03-19T03:15:10.129291+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:36.357+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:10.129236+0000 front 2024-03-19T03:15:10.129277+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:36.357+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:10.129262+0000 front 2024-03-19T03:15:10.129315+0000 (oldest deadline 2024-03-19T03:15:33.029185+0000) 2024-03-19T03:25:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:36.357+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:36.458+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:36.458+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:36.458+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:36.459+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:36.968+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:36.968+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:36.968+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:36.969+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:37.346+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:37.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:37.426+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:37.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:37.426+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:37.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:37.426+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:37.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:37.427+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:38.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:37.995+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:38.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:37.995+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:38.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:37.995+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:38.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:37.996+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:38.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:38.389+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:38.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:38.416+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:38.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:38.416+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:38.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:38.416+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:38.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:38.417+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:39.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:38.985+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:39.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:38.985+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:39.247 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:38.985+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:39.247 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:38.985+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:39.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:39.360+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:39.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:39.439+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:39.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:39.439+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:39.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:39.439+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:39.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:39.440+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:40.327 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:40.022+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:40.327 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:40.022+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:40.327 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:40.022+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:40.327 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:40.023+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:40.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:25:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:25:40.491+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6897 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:25:40.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:40.325+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:40.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:40.420+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:40.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:40.420+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:40.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:40.420+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:40.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:40.421+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:41.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:41.011+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:41.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:41.011+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:41.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:41.011+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:41.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:41.012+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:41.658 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:41.368+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:41.659 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:41.397+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:41.659 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:41.397+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:41.659 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:41.397+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:41.659 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:41.398+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:42.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:42.022+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:42.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:42.022+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:42.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:42.022+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:42.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:42.023+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:42.820 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:42.413+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:42.820 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:42.370+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:42.820 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:42.370+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:42.820 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:42.370+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:42.820 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:42.371+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:43.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:43.034+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:43.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:43.034+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:43.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:43.034+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:43.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:43.035+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:43.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:43.456+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:43.411+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:43.411+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:43.411+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:43.411+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:44.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:43.996+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:43.996+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:43.996+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:43.996+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:44.483+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:44.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:44.412+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.537049+0000 front 2024-03-19T03:15:07.536990+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:44.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:44.412+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.537026+0000 front 2024-03-19T03:15:07.536996+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:44.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:44.412+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.537074+0000 front 2024-03-19T03:15:07.537115+0000 (oldest deadline 2024-03-19T03:15:32.837027+0000) 2024-03-19T03:25:44.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:44.412+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:45.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:44.984+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:45.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:44.984+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:45.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:44.984+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:45.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:44.985+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:45.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:45.488+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:45.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:25:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:25:45.492+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6992 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:25:45.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:45.435+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:46.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:45.982+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:46.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:45.982+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:46.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:45.983+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:46.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:45.983+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:46.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:46.412+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:46.501+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:47.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:47.029+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:47.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:47.029+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:47.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:47.029+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:47.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:47.030+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:47.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:47.540+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:47.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:47.387+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:48.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:48.077+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:48.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:48.077+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:48.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:48.077+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:48.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:48.078+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:48.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:48.570+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:48.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:48.354+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:49.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:49.047+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:49.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:49.047+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:49.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:49.047+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:49.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:49.048+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:49.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:49.608+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:49.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:49.396+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:50.371 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:50.089+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 since back 2024-03-19T03:15:07.970816+0000 front 2024-03-19T03:15:07.970903+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:50.371 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:50.089+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 since back 2024-03-19T03:15:07.970874+0000 front 2024-03-19T03:15:07.970857+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:50.371 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:50.089+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 since back 2024-03-19T03:15:07.970847+0000 front 2024-03-19T03:15:07.970833+0000 (oldest deadline 2024-03-19T03:15:30.870811+0000) 2024-03-19T03:25:50.371 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:50.089+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:50.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:25:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:25:50.493+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7084 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:25:50.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:50.619+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:50.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:50.370+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:51.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:51.084+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:51.659 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:51.579+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:51.659 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:51.386+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:52.387 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:52.121+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:52.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:52.600+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:52.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:52.386+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:53.422 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:53.124+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:53.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:53.643+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:53.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:53.421+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:54.395 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:54.138+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:54.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:54.634+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:54.394+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:55.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:55.170+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:55.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:25:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:25:55.494+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7178 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:25:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:55.434+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:55.613+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:56.566 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:56.176+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:56.567 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:56.399+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:56.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:56.565+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:57.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:57.216+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:57.435+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:58.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:57.598+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:58.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:58.253+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:58.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:58.408+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:59.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:58.638+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:25:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:25:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:25:59.267+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:25:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:25:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:25:59.360+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:00.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:25:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:25:59.670+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:00.318+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:00.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:26:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:26:00.495+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7272 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:26:00.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:00.319+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:01.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:00.629+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:01.324+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:01.325+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:02.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:01.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:01.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:01.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:01.628+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:02.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:02.343+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:02.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:02.302+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:02.626+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:02.626+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:02.626+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:02.627+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:03.551 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:03.301+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:03.552 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:03.280+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:03.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:03.637+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:03.637+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:03.637+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:03.638+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:04.269+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:04.257+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:04.601+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:04.601+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:04.601+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:04.602+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:05.498 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:05.258+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:05.498 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:05.222+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:05.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:26:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:26:05.496+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7368 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:26:05.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:05.639+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:05.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:05.639+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:05.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:05.639+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:05.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:05.640+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:06.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:06.251+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:06.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:06.266+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:07.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:06.666+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:06.666+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:06.666+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:06.667+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:07.291+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:07.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:07.287+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:07.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:07.287+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:07.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:07.287+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:07.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:07.288+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:08.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:07.696+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:07.696+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:07.696+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:07.697+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:08.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:08.246+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:08.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:08.255+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:08.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:08.255+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:08.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:08.255+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:08.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:08.256+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:08.651+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:08.651+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:08.651+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:08.652+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:09.208+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:09.227+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:09.227+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:09.227+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:09.227+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:10.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:09.689+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:10.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:09.689+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:10.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:09.689+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:10.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:09.690+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:10.500 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:10.239+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:10.545 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:10.219+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:10.545 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:10.219+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:10.545 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:10.219+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:10.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:10.220+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:10.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:26:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:26:10.497+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7460 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:26:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:10.702+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:10.702+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:10.702+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:10.703+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:11.243+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:11.171+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:11.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:11.171+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:11.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:11.171+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:11.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:11.172+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:12.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:11.681+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:11.681+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:11.681+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:11.682+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:12.198+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:12.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:12.198+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:12.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:12.198+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:12.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:12.198+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:12.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:12.199+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:13.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:12.632+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:12.632+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:12.632+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:12.633+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:13.540 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:13.165+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:13.540 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:13.165+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:13.540 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:13.165+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:13.540 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:13.166+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:13.541 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:13.183+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:13.541 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:13.183+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:13.541 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:13.183+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:13.541 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:13.183+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:13.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:13.663+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:13.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:13.663+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:13.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:13.663+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:13.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:13.664+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:14.154+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:14.154+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:14.154+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:14.154+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:14.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:14.139+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:14.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:14.139+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:14.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:14.139+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:14.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:14.140+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:14.713+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:14.713+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:14.713+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:14.713+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:15.500 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:15.147+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:15.500 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:15.147+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:15.500 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:15.147+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:15.500 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:15.148+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:15.500 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:15.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:15.500 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:15.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:15.501 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:15.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:15.501 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:15.133+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:15.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:26:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:26:15.498+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7555 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:26:15.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:15.695+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:15.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:15.695+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:15.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:15.695+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:15.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:15.696+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:16.126+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:16.126+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:16.126+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:16.126+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:16.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:16.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:16.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:16.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:16.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:16.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:16.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:16.167+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:16.731+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:16.731+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:16.731+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:16.731+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:17.172+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:17.172+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:17.172+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:17.173+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:17.199+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:17.199+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:17.199+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:17.200+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:18.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:17.736+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:17.736+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:17.736+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:17.737+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:18.200+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:18.200+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:18.200+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:18.201+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:18.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:18.201+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:18.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:18.201+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:18.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:18.201+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:18.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:18.201+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:18.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:18.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:18.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:18.743+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:19.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:19.161+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:19.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:19.161+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:19.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:19.161+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:19.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:19.163+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:19.173+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:19.173+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:19.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:19.173+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:19.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:19.173+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:20.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:19.776+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:20.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:19.776+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:20.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:19.776+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:20.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:19.777+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:20.501 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:20.147+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:20.501 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:20.147+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:20.501 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:20.147+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:20.501 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:20.148+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:20.501 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:20.153+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:20.501 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:20.153+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:20.501 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:20.153+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:20.502 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:20.154+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:20.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:26:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:26:20.499+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7647 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:26:20.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:20.730+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:20.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:20.730+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:20.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:20.730+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:20.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:20.731+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:21.145+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:21.145+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:21.145+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:21.146+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:21.185+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:21.185+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:21.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:21.185+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:21.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:21.185+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:22.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:21.724+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:22.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:21.724+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:22.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:21.724+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:22.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:21.725+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:22.180+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:22.180+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:22.180+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:22.180+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:22.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:22.179+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:22.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:22.179+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:22.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:22.179+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:22.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:22.179+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:22.756+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:22.756+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:22.756+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:22.757+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:23.552 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:23.173+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:23.552 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:23.173+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:23.552 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:23.173+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:23.552 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:23.174+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:23.552 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:23.145+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:23.553 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:23.145+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:23.553 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:23.145+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:23.553 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:23.145+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:23.842 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:23.722+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:23.842 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:23.722+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:23.842 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:23.722+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:23.842 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:23.722+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:24.151+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:24.151+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:24.151+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:24.152+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:24.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:24.126+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:24.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:24.126+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:24.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:24.126+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:24.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:24.126+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:25.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:24.689+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:25.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:24.689+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:25.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:24.689+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:25.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:24.690+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:25.502 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:25.165+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:25.502 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:25.165+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:25.502 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:25.165+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:25.502 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:25.165+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:25.503 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:25.144+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:25.503 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:25.144+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:25.503 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:25.144+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:25.503 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:25.144+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:25.828 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:26:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:26:25.501+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7744 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:26:25.828 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:25.666+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:25.828 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:25.666+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:25.828 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:25.666+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:25.829 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:25.668+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:26.147+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:26.147+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:26.147+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:26.147+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:26.107+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:26.107+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:26.107+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:26.107+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:27.074 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:26.640+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:27.074 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:26.640+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:27.074 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:26.640+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:27.074 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:26.641+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:27.074+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:27.074+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:27.074+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:27.075+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:27.143+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:27.143+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:27.143+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:27.143+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:28.071 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:27.657+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:28.071 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:27.657+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:28.071 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:27.657+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:28.071 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:27.658+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:28.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:28.165+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:28.165+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:28.165+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:28.166+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:28.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:28.070+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:28.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:28.070+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:28.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:28.070+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:28.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:28.071+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:29.074 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:28.632+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:29.074 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:28.632+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:29.074 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:28.632+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:29.074 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:28.633+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:29.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:29.074+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:29.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:29.074+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:29.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:29.074+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:29.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:29.074+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:29.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:29.216+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:29.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:29.216+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:29.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:29.216+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:29.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:29.216+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:30.043 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:29.656+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:30.043 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:29.656+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:30.043 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:29.656+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:30.043 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:29.657+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:30.257+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:30.257+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:30.257+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:30.257+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:30.043+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:30.043+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:30.043+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:30.043+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:30.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:26:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:26:30.503+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7835 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:26:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:30.624+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:30.624+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:30.624+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:30.625+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:31.280 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:31.026+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:31.280 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:31.026+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:31.280 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:31.026+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:31.280 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:31.027+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:31.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:31.279+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:31.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:31.279+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:31.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:31.279+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:31.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:31.281+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:31.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:31.576+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:31.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:31.576+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:31.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:31.576+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:31.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:31.576+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:32.279 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:32.076+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:32.279 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:32.076+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:32.279 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:32.076+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:32.279 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:32.076+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:32.546 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:32.278+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:32.546 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:32.278+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:32.546 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:32.278+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:32.547 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:32.279+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:32.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:32.546+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:32.546+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:32.546+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:32.547+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:33.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:33.239+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:33.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:33.239+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:33.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:33.239+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:33.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:33.239+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:33.081+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:33.081+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:33.081+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:33.081+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:33.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:33.539+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:33.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:33.539+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:33.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:33.539+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:33.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:33.540+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:34.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:34.249+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:34.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:34.249+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:34.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:34.249+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:34.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:34.250+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:34.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:34.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:34.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:34.034+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:34.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:34.564+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:34.564+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:34.564+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:34.566+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:35.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:35.289+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:35.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:35.289+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:35.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:35.289+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:35.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:35.290+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:35.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:35.053+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:35.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:35.053+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:35.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:35.053+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:35.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:35.054+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:35.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:26:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:26:35.503+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7932 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:26:35.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:35.561+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:35.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:35.561+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:35.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:35.561+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:35.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:35.562+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:36.581 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:36.253+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:36.581 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:36.253+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:36.582 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:36.253+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:36.582 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:36.254+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:36.582 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:36.089+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:36.582 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:36.089+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:36.582 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:36.089+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:36.582 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:36.090+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:36.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:36.580+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:36.580+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:36.580+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:36.581+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:37.550 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:37.229+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:37.550 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:37.229+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:37.550 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:37.229+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:37.550 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:37.230+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:37.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:37.101+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:37.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:37.101+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:37.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:37.101+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:37.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:37.102+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:37.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:37.549+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:37.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:37.549+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:37.549+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:37.550+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:38.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:38.109+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:38.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:38.109+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:38.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:38.109+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:38.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:38.110+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:38.563 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:38.274+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:38.563 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:38.274+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:38.563 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:38.274+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:38.563 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:38.275+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:38.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:38.562+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:38.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:38.562+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:38.562+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:38.562+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:39.562 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:39.241+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:39.562 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:39.241+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:39.562 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:39.241+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:39.562 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:39.242+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:39.562 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:39.118+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:39.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:39.118+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:39.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:39.118+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:39.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:39.118+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:39.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:39.561+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:39.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:39.561+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:39.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:39.561+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:39.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:39.562+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:40.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:40.195+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:40.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:40.195+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:40.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:40.195+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:40.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:40.196+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:40.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:40.073+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:40.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:40.073+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:40.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:40.073+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:40.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:40.073+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:40.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:26:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:26:40.504+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8024 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:26:40.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:40.612+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:40.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:40.612+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:40.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:40.612+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:40.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:40.613+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:41.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:41.193+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:41.638 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:41.193+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:41.638 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:41.193+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:41.638 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:41.194+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:41.638 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:41.066+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:41.638 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:41.066+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:41.638 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:41.066+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:41.638 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:41.067+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:41.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:41.568+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:41.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:41.568+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:41.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:41.568+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:41.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:41.569+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:42.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:42.204+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:42.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:42.204+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:42.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:42.204+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:42.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:42.205+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:42.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:42.027+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:42.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:42.027+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:42.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:42.027+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:42.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:42.028+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:42.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:42.577+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:42.577+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:42.577+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:42.577+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:43.240+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:43.240+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:43.240+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:43.240+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:43.060+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:43.060+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:43.060+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:43.061+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:43.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:43.538+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:43.538+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:43.538+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:43.540+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:44.287 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:44.028+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:44.287 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:44.028+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:44.288 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:44.028+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:44.288 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:44.028+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:44.550 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:44.286+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:44.550 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:44.286+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:44.550 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:44.286+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:44.550 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:44.287+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:44.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:44.549+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:44.549+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:44.549+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:44.550+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:45.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:44.986+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:45.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:44.986+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:45.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:44.986+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:45.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:44.987+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:45.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:26:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:26:45.505+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8120 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:26:45.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:45.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:45.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:45.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:45.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:45.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:45.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:45.292+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:45.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:45.512+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:45.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:45.512+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:45.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:45.512+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:45.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:45.515+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:46.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:46.001+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:46.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:46.001+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:46.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:46.001+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:46.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:46.002+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:46.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:46.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:46.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:46.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:46.493+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:46.341+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:46.341+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:46.341+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:46.342+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:47.294 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:46.996+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:47.295 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:46.996+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:47.295 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:46.996+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:47.295 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:46.997+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:47.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:47.293+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:47.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:47.293+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:47.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:47.293+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:47.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:47.294+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:47.525+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:47.525+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:47.525+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:47.525+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:48.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:48.025+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:48.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:48.025+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:48.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:48.025+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:48.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:48.026+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:48.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:48.333+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:48.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:48.333+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:48.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:48.333+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:48.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:48.334+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:48.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:48.535+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:48.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:48.535+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:48.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:48.536+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:48.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:48.536+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:49.335 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:49.058+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:49.336 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:49.058+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:49.336 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:49.058+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:49.336 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:49.059+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:49.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:49.334+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:49.334+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:49.334+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:49.335+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:49.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:49.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:49.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:49.492+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:50.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:50.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:50.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:50.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:50.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:50.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:50.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:50.058+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:50.629 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:26:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:26:50.506+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8212 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:26:50.629 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:50.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:50.630 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:50.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:50.630 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:50.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:50.630 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:50.492+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:50.630 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:50.363+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:50.630 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:50.363+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:50.630 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:50.363+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:50.630 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:50.363+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:51.533 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:51.324+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:51.534 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:51.324+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:51.534 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:51.324+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:51.534 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:51.324+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:51.534 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:51.104+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:51.534 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:51.104+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:51.534 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:51.104+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:51.534 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:51.105+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:51.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:51.532+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:51.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:51.532+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:51.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:51.532+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:51.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:51.532+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:52.490 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:52.322+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:52.490 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:52.322+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:52.490 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:52.322+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:52.490 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:52.323+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:52.491 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:52.123+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:52.491 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:52.123+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:52.491 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:52.123+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:52.491 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:52.123+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:52.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:52.488+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:52.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:52.489+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:52.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:52.489+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:52.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:52.490+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:53.477 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:53.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:53.477 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:53.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:53.477 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:53.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:53.477 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:53.292+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:53.478 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:53.107+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:53.478 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:53.107+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:53.478 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:53.107+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:53.478 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:53.108+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:53.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:53.487+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:53.487+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:53.487+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:53.487+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:54.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:54.063+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:54.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:54.063+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:54.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:54.063+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:54.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:54.064+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:54.267+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:54.267+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:54.267+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:54.268+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:54.513+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:54.513+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:54.513+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:54.514+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:55.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:55.074+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:55.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:55.074+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:55.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:55.074+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:55.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:55.075+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:55.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:55.235+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:55.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:55.235+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:55.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:55.235+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:55.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:55.236+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:55.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:26:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:26:55.507+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8308 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:26:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:55.531+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:55.531+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:55.531+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:55.532+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:56.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:56.263+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:56.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:56.263+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:56.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:56.263+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:56.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:56.264+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:56.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:56.087+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:56.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:56.087+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:56.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:56.087+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:56.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:56.088+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:56.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:56.486+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:56.486+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:56.486+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:56.487+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:57.038+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:57.038+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:57.038+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:57.039+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:57.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:57.225+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:57.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:57.225+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:57.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:57.226+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:57.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:57.226+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:57.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:57.482+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:57.483+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:57.483+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:57.484+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:58.052+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:58.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:58.052+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:58.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:58.052+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:58.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:58.052+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:58.200+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:58.200+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:58.200+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:58.201+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:58.443+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:58.443+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:58.443+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:58.444+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:26:59.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:59.240+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:59.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:59.240+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:59.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:59.240+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:26:59.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:26:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:26:59.241+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:59.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:59.015+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:59.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:59.015+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:59.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:59.015+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:26:59.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:26:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:26:59.016+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:26:59.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:59.463+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:59.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:59.463+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:59.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:59.463+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:26:59.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:26:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:26:59.463+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:00.208 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:00.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:00.209 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:00.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:00.209 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:00.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:00.209 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:00.058+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:00.509 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:00.207+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:00.510 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:00.207+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:00.510 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:00.207+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:00.510 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:00.208+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:00.510 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:00.434+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:00.510 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:00.434+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:00.510 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:00.434+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:00.510 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:00.435+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:00.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:27:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:27:00.508+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8400 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:27:01.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:01.232+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:01.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:01.232+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:01.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:01.232+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:01.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:01.232+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:01.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:01.053+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:01.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:01.053+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:01.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:01.053+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:01.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:01.054+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:01.665 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:01.412+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:01.665 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:01.412+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:01.665 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:01.413+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:01.665 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:01.413+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:02.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:02.202+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:02.202+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:02.202+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:02.202+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:02.081+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:02.081+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:02.081+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:02.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:02.082+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:02.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:02.404+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:02.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:02.404+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:02.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:02.404+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:02.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:02.404+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:03.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:03.158+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:03.158+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:03.158+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:03.158+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:03.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:03.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:03.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:03.058+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:03.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:03.357+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:03.357+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:03.357+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:03.358+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:04.190+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:04.190+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:04.190+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:04.190+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:04.097+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:04.097+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:04.097+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:04.098+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:04.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:04.342+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:04.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:04.342+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:04.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:04.342+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:04.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:04.343+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:05.386 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:05.184+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:05.386 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:05.184+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:05.386 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:05.184+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:05.386 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:05.185+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:05.387 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:05.122+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:05.387 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:05.122+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:05.387 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:05.122+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:05.387 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:05.123+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:05.764 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:27:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:27:05.509+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8496 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:27:05.764 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:05.385+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:05.764 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:05.385+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:05.764 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:05.385+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:05.764 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:05.386+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:06.419 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:06.163+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:06.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:06.163+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:06.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:06.163+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:06.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:06.163+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:06.420 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:06.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:06.420 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:06.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:06.420 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:06.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:06.420 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:06.168+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:06.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:06.418+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:06.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:06.418+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:06.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:06.418+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:06.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:06.419+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:07.167+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:07.167+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:07.167+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:07.167+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:07.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:07.370+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:07.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:07.370+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:07.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:07.370+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:07.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:07.371+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:07.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:07.160+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:07.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:07.160+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:07.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:07.160+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:07.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:07.160+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:08.401 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:08.120+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:08.402 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:08.120+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:08.402 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:08.120+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:08.402 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:08.120+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:08.402 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:08.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:08.402 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:08.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:08.402 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:08.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:08.402 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:08.138+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:08.836 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:08.400+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:08.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:08.400+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:08.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:08.400+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:08.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:08.401+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:09.407 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:09.115+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:09.407 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:09.115+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:09.407 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:09.115+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:09.407 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:09.116+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:09.407 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:09.123+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:09.407 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:09.123+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:09.407 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:09.123+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:09.407 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:09.124+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:09.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:09.405+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:09.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:09.405+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:09.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:09.405+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:09.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:09.406+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:10.376 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:10.124+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:10.376 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:10.124+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:10.377 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:10.124+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:10.377 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:10.124+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:10.377 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:10.120+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:10.377 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:10.120+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:10.377 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:10.120+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:10.377 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:10.121+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:10.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:27:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:27:10.510+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8588 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:27:10.636 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:10.375+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:10.636 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:10.375+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:10.636 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:10.375+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:10.636 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:10.376+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:11.381 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:11.148+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:11.381 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:11.148+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:11.381 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:11.148+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:11.381 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:11.149+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:11.381 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:11.097+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:11.381 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:11.097+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:11.381 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:11.097+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:11.381 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:11.098+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:11.665 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:11.379+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:11.665 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:11.379+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:11.665 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:11.379+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:11.665 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:11.380+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:12.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:12.056+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:12.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:12.056+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:12.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:12.056+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:12.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:12.056+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:12.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:12.163+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:12.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:12.163+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:12.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:12.163+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:12.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:12.163+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:12.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:12.361+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:12.361+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:12.361+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:12.362+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:13.333 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:13.062+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:13.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:13.062+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:13.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:13.062+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:13.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:13.063+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:13.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:13.190+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:13.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:13.190+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:13.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:13.190+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:13.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:13.191+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:13.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:13.332+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:13.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:13.332+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:13.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:13.332+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:13.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:13.333+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:14.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:14.225+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:14.225+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:14.225+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:14.226+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:14.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:14.037+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:14.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:14.037+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:14.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:14.037+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:14.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:14.038+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:14.369+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:14.369+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:14.369+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:14.370+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:15.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:15.214+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:15.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:15.214+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:15.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:15.214+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:15.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:15.214+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:15.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:15.058+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:15.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:15.058+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:15.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:15.058+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:15.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:15.058+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:15.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:27:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:27:15.512+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8685 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:27:15.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:15.396+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:15.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:15.396+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:15.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:15.396+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:15.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:15.397+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:16.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:16.260+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:16.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:16.260+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:16.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:16.260+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:16.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:16.260+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:16.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:16.070+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:16.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:16.070+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:16.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:16.070+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:16.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:16.070+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:16.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:16.437+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:16.437+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:16.437+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:16.438+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:17.286 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:17.024+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:17.286 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:17.024+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:17.286 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:17.024+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:17.286 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:17.024+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:17.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:17.284+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:17.284+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:17.284+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:17.285+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:17.412+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:17.412+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:17.412+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:17.413+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:18.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:18.241+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:18.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:18.241+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:18.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:18.241+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:18.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:18.242+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:18.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:18.047+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:18.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:18.047+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:18.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:18.047+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:18.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:18.047+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:18.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:18.370+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:18.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:18.370+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:18.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:18.370+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:18.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:18.371+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:19.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:19.199+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:19.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:19.199+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:19.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:19.199+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:19.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:19.200+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:19.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:19.049+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:19.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:19.049+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:19.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:19.049+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:19.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:19.050+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:19.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:19.341+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:19.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:19.341+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:19.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:19.341+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:19.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:19.342+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:20.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:20.200+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:20.200+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:20.200+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:20.201+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:20.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:20.071+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:20.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:20.071+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:20.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:20.071+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:20.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:20.072+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:20.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:20.297+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:20.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:20.297+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:20.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:20.298+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:20.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:20.298+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:20.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:27:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:27:20.513+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8777 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:27:21.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:21.166+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:21.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:21.166+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:21.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:21.166+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:21.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:21.166+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:21.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:21.291+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:21.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:21.291+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:21.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:21.291+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:21.341 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:21.291+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:21.341 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:21.082+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:21.341 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:21.082+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:21.341 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:21.082+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:21.342 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:21.083+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:22.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:22.162+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:22.162+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:22.162+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:22.163+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:22.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:22.324+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:22.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:22.324+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:22.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:22.324+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:22.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:22.324+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:22.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:22.081+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:22.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:22.081+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:22.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:22.081+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:22.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:22.081+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:23.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:23.144+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:23.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:23.144+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:23.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:23.144+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:23.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:23.144+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:23.284+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:23.284+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:23.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:23.284+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:23.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:23.285+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:23.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:23.050+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:23.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:23.050+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:23.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:23.050+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:23.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:23.051+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:24.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:24.265+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:24.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:24.265+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:24.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:24.265+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:24.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:24.266+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:24.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:24.125+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:24.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:24.125+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:24.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:24.125+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:24.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:24.127+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:24.341 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:24.016+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:24.341 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:24.016+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:24.341 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:24.016+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:24.341 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:24.017+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:25.303 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:25.129+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:25.304 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:25.129+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:25.304 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:25.129+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:25.304 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:25.129+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:25.304 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:25.008+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:25.304 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:25.008+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:25.304 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:25.008+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:25.304 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:25.009+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:25.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:27:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:27:25.514+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8873 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:27:25.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:25.302+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:25.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:25.302+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:25.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:25.302+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:25.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:25.303+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:26.302 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:26.301+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:26.303 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:26.112+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:26.303 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:26.112+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:26.303 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:26.112+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:26.303 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:26.113+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:26.303 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:26.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:26.303 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:26.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:26.303 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:26.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:26.303 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:26.051+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:26.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:26.301+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:26.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:26.301+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:26.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:26.302+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:27.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:27.261+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:27.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:27.261+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:27.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:27.261+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:27.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:27.262+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:27.111+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:27.111+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:27.111+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:27.112+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:27.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:27.039+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:27.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:27.039+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:27.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:27.039+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:27.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:27.040+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:28.342 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:28.250+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:28.342 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:28.250+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:28.342 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:28.250+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:28.343 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:28.251+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:28.343 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:28.081+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:28.343 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:28.081+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:28.343 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:28.081+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:28.343 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:28.082+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:28.344 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:28.086+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:28.344 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:28.086+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:28.344 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:28.086+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:28.344 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:28.087+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:29.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:29.090+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:29.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:29.090+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:29.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:29.090+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:29.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:29.091+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:29.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:29.077+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:29.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:29.077+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:29.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:29.077+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:29.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:29.078+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:29.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:29.259+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:29.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:29.259+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:29.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:29.259+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:29.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:29.260+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:30.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:30.289+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:30.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:30.289+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:30.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:30.289+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:30.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:30.290+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:30.084+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:30.084+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:30.084+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:30.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:30.085+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:30.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:30.077+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:30.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:30.077+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:30.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:30.077+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:30.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:30.077+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:30.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:27:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:27:30.515+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8965 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:27:31.291 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:31.061+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:31.292 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:31.061+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:31.292 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:31.061+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:31.292 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:31.062+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:31.292 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:31.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:31.292 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:31.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:31.292 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:31.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:31.292 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:31.052+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:31.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:31.290+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:31.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:31.290+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:31.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:31.290+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:31.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:31.291+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:32.337 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:32.031+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:32.337 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:32.031+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:32.337 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:32.031+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:32.337 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:32.032+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:32.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:32.039+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:32.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:32.039+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:32.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:32.039+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:32.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:32.040+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:32.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:32.335+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:32.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:32.335+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:32.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:32.335+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:32.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:32.336+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:33.333 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:33.034+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:33.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:33.034+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:33.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:33.034+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:33.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:33.035+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:33.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:33.084+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:33.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:33.084+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:33.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:33.084+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:33.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:33.085+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:33.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:33.332+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:33.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:33.332+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:33.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:33.332+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:33.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:33.333+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:34.315 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:34.020+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:34.315 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:34.020+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:34.315 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:34.020+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:34.315 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:34.021+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:34.315 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:34.103+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:34.315 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:34.103+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:34.315 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:34.103+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:34.315 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:34.104+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:34.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:34.313+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:34.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:34.313+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:34.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:34.313+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:34.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:34.314+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:35.336 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:35.064+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:35.336 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:35.064+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:35.336 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:35.064+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:35.336 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:35.065+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:35.336 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:35.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:35.336 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:35.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:35.336 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:35.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:35.336 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:35.107+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:35.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:27:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:27:35.516+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9061 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:27:35.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:35.335+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:35.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:35.335+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:35.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:35.335+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:35.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:35.336+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:36.308 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:36.027+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:36.308 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:36.027+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:36.308 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:36.027+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:36.309 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:36.028+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:36.309 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:36.115+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:36.309 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:36.115+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:36.309 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:36.115+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:36.309 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:36.115+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:36.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:36.307+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:36.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:36.307+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:36.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:36.307+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:36.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:36.308+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:37.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:37.305+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:37.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:37.305+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:37.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:37.305+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:37.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:37.306+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:37.059+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:37.059+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:37.059+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:37.060+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:37.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:37.105+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:37.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:37.105+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:37.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:37.105+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:37.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:37.106+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:38.291 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:38.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:38.291 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:38.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:38.291 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:38.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:38.292 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:38.133+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:38.292 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:38.022+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:38.292 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:38.022+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:38.292 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:38.022+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:38.292 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:38.023+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:38.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:38.290+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:38.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:38.290+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:38.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:38.290+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:38.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:38.291+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:39.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:38.987+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:39.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:38.987+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:39.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:38.987+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:39.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:38.988+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:39.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:39.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:39.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:39.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:39.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:39.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:39.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:39.163+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:39.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:39.289+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:39.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:39.289+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:39.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:39.289+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:39.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:39.290+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:40.319 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:40.000+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:40.319 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:40.000+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:40.319 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:40.000+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:40.319 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:40.001+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:40.319 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:40.143+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:40.320 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:40.143+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:40.320 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:40.143+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:40.320 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:40.144+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:40.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:27:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:27:40.517+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9149 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:27:40.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:40.318+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:40.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:40.318+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:40.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:40.318+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:40.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:40.319+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:41.304 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:40.973+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:41.304 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:40.973+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:41.305 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:40.973+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:41.305 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:40.974+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:41.305 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:41.113+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:41.305 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:41.113+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:41.305 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:41.113+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:41.305 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:41.114+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:41.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:41.303+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:41.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:41.303+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:41.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:41.303+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:41.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:41.304+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:42.270 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:42.155+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:42.270 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:42.155+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:42.270 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:42.155+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:42.270 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:42.156+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:42.270 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:41.990+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:42.270 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:41.990+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:42.271 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:41.991+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:42.271 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:41.991+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:42.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:42.268+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:42.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:42.268+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:42.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:42.268+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:42.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:42.269+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:43.138+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:43.138+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:43.138+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:43.139+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:43.008+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:43.008+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:43.008+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:43.009+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:43.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:43.254+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:43.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:43.254+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:43.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:43.254+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:43.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:43.255+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:44.225+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:44.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:44.225+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:44.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:44.225+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:44.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:44.226+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:44.011+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:44.011+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:44.011+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:44.011+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:44.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:44.172+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:44.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:44.172+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:44.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:44.172+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:44.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:44.173+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:45.270 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:45.008+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:45.271 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:45.008+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:45.271 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:45.008+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:45.271 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:45.008+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:45.271 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:45.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:45.271 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:45.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:45.271 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:45.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:45.271 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:45.168+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:45.531 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:27:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:27:45.519+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9238 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:27:45.531 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:45.270+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:45.532 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:45.270+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:45.532 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:45.270+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:45.532 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:45.270+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:46.244 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:45.972+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:46.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:45.972+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:46.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:45.973+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:46.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:45.973+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:46.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:46.151+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:46.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:46.151+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:46.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:46.151+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:46.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:46.152+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:46.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:46.243+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:46.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:46.243+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:46.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:46.243+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:46.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:46.245+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:47.283 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:47.023+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:47.283 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:47.023+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:47.284 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:47.023+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:47.284 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:47.024+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:47.284 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:47.151+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:47.284 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:47.151+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:47.284 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:47.151+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:47.284 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:47.152+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:47.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:47.283+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:47.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:47.283+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:47.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:47.283+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:47.284+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:48.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:48.071+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:48.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:48.071+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:48.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:48.071+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:48.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:48.071+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:48.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:48.235+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:48.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:48.235+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:48.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:48.235+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:48.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:48.235+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:48.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:48.151+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:48.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:48.151+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:48.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:48.151+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:48.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:48.152+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:49.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:49.235+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:49.235+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:49.235+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:49.236+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:49.089+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:49.089+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:49.089+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:49.090+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:49.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:49.155+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:49.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:49.155+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:49.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:49.155+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:49.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:49.156+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:50.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:50.053+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:50.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:50.053+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:50.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:50.053+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:50.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:50.054+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:50.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:50.230+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:50.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:50.230+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:50.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:50.230+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:50.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:50.231+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:50.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:50.153+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:50.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:50.153+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:50.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:50.153+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:50.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:50.154+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:50.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:27:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:27:50.521+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9320 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:27:51.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:51.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:51.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:51.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:51.138+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:51.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:51.068+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:51.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:51.068+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:51.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:51.068+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:51.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:51.069+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:51.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:51.188+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:51.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:51.188+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:51.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:51.188+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:51.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:51.189+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:52.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:52.213+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:52.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:52.213+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:52.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:52.213+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:52.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:52.214+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:52.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:52.029+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:52.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:52.029+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:52.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:52.029+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:52.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:52.030+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:52.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:52.100+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:52.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:52.100+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:52.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:52.100+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:52.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:52.100+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:53.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:53.011+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:53.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:53.011+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:53.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:53.011+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:53.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:53.012+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:53.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:53.186+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:53.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:53.186+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:53.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:53.186+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:53.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:53.187+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:53.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:53.125+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:53.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:53.125+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:53.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:53.125+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:53.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:53.126+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:54.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:54.135+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:54.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:54.135+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:54.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:54.135+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:54.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:54.136+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:54.055+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:54.055+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:54.055+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:54.056+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:54.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:54.196+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:54.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:54.196+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:54.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:54.196+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:54.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:54.196+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:55.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:55.067+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:55.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:55.067+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:55.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:55.067+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:55.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:55.068+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:55.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:55.157+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:55.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:55.157+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:55.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:55.157+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:55.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:55.159+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:55.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:55.120+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:55.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:55.120+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:55.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:55.120+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:55.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:55.121+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:55.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:27:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:27:55.521+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9407 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:27:56.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:56.030+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:56.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:56.030+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:56.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:56.030+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:56.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:56.032+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:56.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:56.119+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:56.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:56.119+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:56.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:56.119+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:56.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:56.120+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:56.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:56.145+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:56.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:56.145+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:56.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:56.145+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:56.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:56.146+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:57.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:57.060+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:57.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:57.060+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:57.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:57.060+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:57.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:57.060+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:57.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:57.138+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:57.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:57.138+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:57.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:57.138+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:57.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:57.139+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:57.115+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:57.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:57.115+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:57.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:57.115+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:57.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:57.115+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:58.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:58.038+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:58.038+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:58.038+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:58.039+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:58.095+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:58.095+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:58.095+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:58.096+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:27:58.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:58.126+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:58.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:58.126+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:58.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:58.126+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:58.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:58.127+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:59.117+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:59.117+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:59.117+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:27:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:27:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:27:59.118+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:59.088+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:59.088+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:59.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:59.088+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:27:59.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:27:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:27:59.089+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:27:59.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:59.106+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:59.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:59.106+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:59.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:59.106+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:27:59.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:27:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:27:59.107+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:00.525 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:00.092+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:00.525 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:00.092+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:00.525 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:00.092+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:00.526 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:00.092+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:00.526 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:00.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:00.526 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:00.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:00.526 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:00.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:00.526 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:00.137+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:00.526 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:00.090+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:00.526 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:00.090+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:00.526 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:00.090+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:00.527 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:00.091+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:00.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:28:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:28:00.523+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9490 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:28:01.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:01.185+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:01.185+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:01.185+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:01.185+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:01.129+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:01.129+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:01.129+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:01.130+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:01.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:01.102+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:01.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:01.102+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:01.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:01.102+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:01.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:01.103+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:02.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:02.134+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:02.134+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:02.134+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:02.135+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:02.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:02.071+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:02.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:02.071+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:02.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:02.071+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:02.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:02.072+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:02.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:02.160+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:02.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:02.160+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:02.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:02.160+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:02.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:02.161+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:03.140+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:03.140+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:03.140+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:03.141+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:03.167+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:03.167+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:03.167+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:03.168+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:03.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:03.023+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:03.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:03.023+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:03.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:03.023+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:03.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:03.023+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:04.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:04.115+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:04.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:04.115+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:04.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:04.115+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:04.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:04.116+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:04.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:04.212+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:04.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:04.212+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:04.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:04.212+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:04.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:04.214+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:04.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:04.070+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:04.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:04.070+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:04.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:04.070+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:04.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:04.070+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:05.189+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:05.190+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:05.190+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:05.190+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:05.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:05.133+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:05.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:05.133+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:05.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:05.133+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:05.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:05.134+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:05.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:05.068+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:05.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:05.068+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:05.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:05.068+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:05.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:05.068+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:05.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:28:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:28:05.524+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9580 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:28:06.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:06.070+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:06.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:06.070+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:06.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:06.070+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:06.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:06.071+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:06.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:06.212+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:06.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:06.212+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:06.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:06.212+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:06.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:06.212+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:06.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:06.094+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:06.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:06.094+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:06.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:06.094+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:06.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:06.095+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:07.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:07.179+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:07.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:07.179+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:07.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:07.179+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:07.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:07.180+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:07.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:07.039+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:07.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:07.039+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:07.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:07.039+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:07.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:07.040+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:07.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:07.105+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:07.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:07.105+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:07.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:07.105+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:07.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:07.106+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:08.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:08.181+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:08.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:08.181+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:08.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:08.181+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:08.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:08.181+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:08.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:08.023+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:08.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:08.023+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:08.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:08.023+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:08.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:08.024+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:08.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:08.117+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:08.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:08.117+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:08.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:08.117+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:08.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:08.117+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:09.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:09.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:09.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:09.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:09.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:09.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:09.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:09.022+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:09.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:09.206+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:09.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:09.206+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:09.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:09.206+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:09.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:09.206+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:09.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:09.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:09.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:09.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:09.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:09.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:09.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:09.107+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:10.277 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:09.997+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:10.277 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:09.997+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:10.277 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:09.997+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:10.277 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:09.997+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:10.277 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:10.206+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:10.277 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:10.206+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:10.277 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:10.206+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:10.278 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:10.206+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:10.278 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:10.140+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:10.278 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:10.140+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:10.278 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:10.140+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:10.278 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:10.141+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:10.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:28:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:28:10.525+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9669 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:28:11.207 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:11.152+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:11.207 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:11.152+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:11.207 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:11.152+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:11.207 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:11.153+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:11.208 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:10.949+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:11.208 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:10.949+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:11.208 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:10.949+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:11.208 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:10.950+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:11.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:11.206+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:11.206+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:11.206+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:11.207+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:12.226 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:11.964+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:12.226 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:11.964+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:12.226 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:11.964+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:12.226 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:11.964+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:12.227 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:12.181+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:12.227 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:12.181+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:12.227 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:12.181+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:12.227 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:12.182+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:12.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:12.225+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:12.225+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:12.225+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:12.226+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:13.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:12.942+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:13.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:12.942+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:13.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:12.942+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:13.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:12.942+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:13.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:13.199+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:13.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:13.199+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:13.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:13.199+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:13.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:13.200+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:13.560 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:13.195+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:13.560 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:13.195+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:13.560 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:13.195+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:13.560 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:13.196+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:14.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:13.911+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:14.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:13.911+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:14.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:13.911+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:14.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:13.912+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:14.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:14.199+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:14.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:14.199+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:14.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:14.199+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:14.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:14.200+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:14.214+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:14.214+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:14.214+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:14.215+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:15.192 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:14.882+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:15.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:14.882+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:15.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:14.882+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:15.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:14.883+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:15.528 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:15.217+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:15.528 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:15.217+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:15.528 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:15.217+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:15.528 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:15.218+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:15.528 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:15.191+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:15.529 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:15.191+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:15.529 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:15.191+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:15.529 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:15.192+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:15.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:28:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:28:15.527+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9765 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:28:16.149 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:15.862+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:16.149 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:15.862+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:16.149 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:15.862+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:16.149 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:15.863+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:16.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:16.178+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:16.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:16.178+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:16.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:16.178+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:16.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:16.178+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:16.421 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:16.148+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:16.421 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:16.148+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:16.421 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:16.148+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:16.421 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:16.149+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:17.168 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:17.134+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:17.169 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:17.134+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:17.169 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:17.134+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:17.169 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:17.135+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:17.169 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:16.903+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:17.169 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:16.903+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:17.169 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:16.903+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:17.169 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:16.904+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:17.170 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:17.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:17.170 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:17.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:17.170 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:17.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:17.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:17.168+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:18.144 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:17.879+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:18.144 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:17.879+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:18.144 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:17.879+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:18.144 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:17.880+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:18.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:18.158+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:18.158+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:18.158+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:18.158+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:18.143+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:18.143+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:18.143+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:18.144+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:19.152 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:18.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:19.152 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:18.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:19.152 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:18.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:19.152 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:18.866+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:19.153 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:19.113+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:19.153 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:19.113+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:19.153 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:19.113+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:19.153 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:19.114+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:19.151+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:19.151+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:19.151+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:19.152+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:20.124 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:19.861+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:20.124 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:19.861+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:20.124 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:19.861+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:20.124 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:19.861+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:20.125 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:20.110+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:20.125 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:20.110+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:20.125 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:20.110+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:20.125 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:20.110+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:20.529 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:20.123+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:20.529 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:20.123+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:20.529 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:20.123+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:20.529 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:20.124+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:20.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:28:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:28:20.528+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9857 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:28:21.151 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:20.849+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:21.151 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:20.849+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:21.151 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:20.849+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:21.151 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:20.850+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:21.152 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:21.087+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:21.152 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:21.087+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:21.152 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:21.087+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:21.152 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:21.088+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:21.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:21.150+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:21.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:21.150+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:21.150+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:21.150+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:22.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:21.839+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:22.063 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:21.839+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:22.063 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:21.839+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:22.063 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:21.840+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:22.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:22.166+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:22.166+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:22.166+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:22.167+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:22.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:22.061+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:22.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:22.061+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:22.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:22.061+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:22.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:22.062+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:23.146 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:22.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:23.146 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:22.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:23.146 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:22.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:23.146 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:22.848+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:23.146 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:23.035+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:23.146 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:23.035+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:23.147 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:23.035+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:23.147 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:23.036+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:23.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:23.144+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:23.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:23.144+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:23.560 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:23.145+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:23.560 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:23.146+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:24.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:23.832+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:24.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:23.832+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:24.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:23.832+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:24.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:23.833+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:24.042+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:24.042+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:24.042+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:24.042+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:24.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:24.111+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:24.111+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:24.111+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:24.112+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:25.133 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:24.881+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:25.133 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:24.881+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:25.133 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:24.881+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:25.133 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:24.882+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:25.133 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:25.131+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:25.134 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:25.131+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:25.134 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:25.131+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:25.134 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:25.132+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:25.134 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:25.091+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:25.134 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:25.091+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:25.134 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:25.091+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:25.134 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:25.092+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:25.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:28:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:28:25.529+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9953 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:28:26.138 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:26.113+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:26.138 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:26.113+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:26.138 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:26.113+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:26.138 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:26.114+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:26.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:25.869+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:26.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:25.869+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:26.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:25.869+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:26.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:25.870+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:26.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:26.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:26.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:26.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:26.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:26.138+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:27.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:26.917+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:27.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:26.917+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:27.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:26.917+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:27.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:26.918+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:27.091+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:27.091+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:27.091+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:27.092+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:27.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:27.101+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:27.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:27.102+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:27.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:27.102+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:27.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:27.102+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:28.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:28.085+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:28.085+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:28.085+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:28.086+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:28.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:27.955+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:28.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:27.955+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:28.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:27.955+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:28.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:27.956+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:28.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:28.098+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:28.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:28.098+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:28.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:28.098+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:28.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:28.099+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:29.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:29.124+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:29.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:29.124+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:29.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:29.124+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:29.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:29.125+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:29.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:28.983+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:29.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:28.983+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:29.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:28.983+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:29.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:28.984+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:29.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:29.063+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:29.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:29.063+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:29.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:29.063+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:29.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:29.064+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:30.152+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:30.152+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:30.152+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:30.154+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:30.037+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:30.037+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:30.037+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:30.038+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:30.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:30.032+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:30.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:30.032+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:30.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:30.032+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:30.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:30.032+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:30.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:28:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:28:30.531+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10046 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:28:31.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:31.179+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:31.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:31.179+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:31.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:31.179+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:31.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:31.180+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:31.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:31.029+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:31.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:31.029+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:31.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:31.029+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:31.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:31.030+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:31.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:31.057+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:31.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:31.057+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:31.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:31.057+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:31.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:31.057+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:32.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:32.167+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:32.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:32.167+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:32.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:32.167+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:32.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:32.168+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:32.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:32.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:32.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:32.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:32.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:32.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:32.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:32.052+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:32.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:32.016+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:32.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:32.016+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:32.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:32.016+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:32.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:32.016+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:33.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:33.025+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:33.026+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:33.026+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:33.026+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:33.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:33.119+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:33.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:33.119+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:33.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:33.119+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:33.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:33.120+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:33.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:33.034+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:33.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:33.034+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:33.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:33.034+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:33.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:33.034+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:34.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:33.987+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:33.987+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:33.987+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:33.988+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:34.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:34.136+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:34.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:34.136+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:34.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:34.136+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:34.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:34.136+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:34.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:34.024+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:34.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:34.024+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:34.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:34.024+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:34.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:34.025+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:35.116 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:35.026+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:35.116 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:35.026+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:35.116 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:35.026+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:35.116 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:35.027+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:35.117 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:34.990+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:35.117 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:34.990+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:35.117 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:34.990+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:35.117 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:34.991+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:35.533 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:35.115+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:35.533 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:35.115+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:35.533 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:35.115+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:35.533 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:35.115+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:35.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:28:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:28:35.532+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10143 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:28:36.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:36.160+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:36.160+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:36.160+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:36.162+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:36.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:36.050+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:36.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:36.050+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:36.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:36.050+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:36.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:36.051+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:36.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:35.986+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:36.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:35.986+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:36.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:35.986+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:36.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:35.987+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:37.207 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:36.949+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:37.208 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:36.949+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:37.208 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:36.949+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:37.208 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:36.949+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:37.208 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:37.006+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:37.208 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:37.006+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:37.208 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:37.006+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:37.208 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:37.007+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:37.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:37.206+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:37.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:37.206+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:37.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:37.206+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:37.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:37.207+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:38.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:38.173+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:38.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:38.173+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:38.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:38.173+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:38.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:38.174+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:38.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:37.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:38.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:37.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:38.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:37.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:38.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:37.930+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:38.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:37.991+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:38.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:37.991+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:38.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:37.991+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:38.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:37.992+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:39.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:39.155+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:39.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:39.155+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:39.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:39.155+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:39.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:39.156+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:39.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:39.028+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:39.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:39.028+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:39.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:39.028+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:39.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:39.030+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:39.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:38.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:39.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:38.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:39.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:38.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:39.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:38.906+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:40.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:40.149+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:40.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:40.149+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:40.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:40.149+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:40.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:40.150+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:40.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:39.984+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:40.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:39.984+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:40.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:39.984+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:40.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:39.985+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:40.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:39.900+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:40.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:39.900+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:40.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:39.900+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:40.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:39.901+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:40.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:28:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:28:40.533+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10236 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:28:41.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:41.131+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:41.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:41.131+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:41.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:41.131+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:41.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:41.132+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:41.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:40.959+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:41.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:40.959+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:41.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:40.959+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:41.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:40.959+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:41.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:40.899+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:41.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:40.899+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:41.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:40.899+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:41.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:40.900+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:41.983 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:41.879+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:41.983 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:41.879+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:41.983 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:41.879+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:41.984 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:41.880+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:41.982+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:42.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:41.982+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:42.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:41.982+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:42.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:41.983+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:42.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:42.179+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:42.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:42.179+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:42.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:42.179+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:42.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:42.179+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:43.187 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:43.019+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:43.187 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:43.019+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:43.187 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:43.019+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:43.187 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:43.020+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:43.187 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:42.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:43.187 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:42.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:43.187 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:42.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:43.188 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:42.930+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:43.561 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:43.185+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:43.561 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:43.185+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:43.561 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:43.185+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:43.561 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:43.186+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:44.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:44.172+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:44.172+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:44.172+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:44.173+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:43.948+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:43.948+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:43.948+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:43.949+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:44.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:44.003+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:44.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:44.003+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:44.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:44.003+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:44.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:44.004+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:45.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:45.033+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:45.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:45.033+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:45.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:45.033+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:45.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:45.033+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:45.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:45.131+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:45.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:45.131+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:45.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:45.131+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:45.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:45.131+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:45.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:44.980+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:45.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:44.980+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:45.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:44.980+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:45.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:44.980+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:45.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:28:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:28:45.534+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10333 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:28:46.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:46.136+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:46.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:46.136+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:46.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:46.136+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:46.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:46.137+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:46.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:46.083+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:46.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:46.083+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:46.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:46.083+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:46.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:46.084+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:46.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:45.964+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:46.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:45.964+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:46.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:45.964+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:46.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:45.965+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:47.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:47.145+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:47.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:47.145+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:47.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:47.145+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:47.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:47.146+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:47.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:47.120+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:47.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:47.120+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:47.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:47.120+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:47.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:47.121+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:47.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:47.004+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:47.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:47.004+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:47.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:47.004+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:47.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:47.005+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:48.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:48.022+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:48.022+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:48.022+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:48.022+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:48.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:48.135+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:48.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:48.135+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:48.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:48.135+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:48.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:48.136+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:48.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:48.150+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:48.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:48.150+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:48.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:48.150+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:48.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:48.151+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:49.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:49.054+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:49.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:49.054+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:49.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:49.054+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:49.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:49.054+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:49.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:49.097+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:49.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:49.097+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:49.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:49.097+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:49.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:49.098+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:49.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:49.173+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:49.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:49.173+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:49.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:49.173+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:49.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:49.174+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:50.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:50.189+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:50.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:50.189+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:50.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:50.189+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:50.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:50.190+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:50.341 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:50.132+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:50.341 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:50.132+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:50.341 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:50.132+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:50.341 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:50.132+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:50.341 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:50.069+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:50.341 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:50.069+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:50.341 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:50.069+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:50.341 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:50.070+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:50.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:28:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:28:50.536+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10426 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:28:51.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:51.038+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:51.038+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:51.038+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:51.039+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:51.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:51.139+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:51.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:51.139+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:51.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:51.139+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:51.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:51.140+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:51.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:51.152+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:51.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:51.152+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:51.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:51.152+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:51.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:51.152+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:52.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:52.075+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:52.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:52.075+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:52.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:52.075+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:52.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:52.076+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:52.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:52.144+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:52.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:52.144+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:52.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:52.144+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:52.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:52.145+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:52.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:52.163+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:52.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:52.163+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:52.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:52.163+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:52.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:52.164+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:53.561 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:53.192+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:53.561 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:53.192+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:53.562 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:53.192+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:53.562 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:53.193+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:53.562 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:53.174+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:53.562 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:53.174+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:53.562 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:53.174+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:53.562 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:53.176+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:53.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:53.110+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:53.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:53.110+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:53.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:53.110+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:53.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:53.110+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:54.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:54.188+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:54.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:54.188+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:54.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:54.188+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:54.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:54.189+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:54.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:54.210+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:54.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:54.210+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:54.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:54.210+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:54.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:54.211+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:54.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:54.138+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:54.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:54.138+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:54.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:54.138+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:54.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:54.139+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:55.538 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:55.169+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:55.538 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:55.169+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:55.538 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:55.169+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:55.538 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:55.170+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:55.539 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:55.220+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:55.539 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:55.220+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:55.539 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:55.220+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:55.539 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:55.220+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:55.539 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:55.177+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:55.539 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:55.177+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:55.539 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:55.177+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:55.539 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:55.178+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:55.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:28:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:28:55.537+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10521 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:28:56.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:56.218+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:56.218+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:56.218+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:56.218+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:56.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:56.243+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:56.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:56.243+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:56.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:56.243+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:56.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:56.244+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:56.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:56.208+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:56.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:56.208+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:56.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:56.208+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:56.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:56.209+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:57.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:57.187+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:57.187+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:57.187+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:57.187+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:57.196+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:57.196+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:57.196+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:57.197+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:57.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:57.242+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:57.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:57.242+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:57.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:57.242+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:57.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:57.243+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:58.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:58.199+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:58.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:58.199+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:58.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:58.199+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:58.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:58.200+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:58.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:58.227+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:58.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:58.227+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:58.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:58.227+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:58.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:58.227+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:58.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:58.224+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:58.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:58.224+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:58.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:58.224+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:58.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:58.225+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:59.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:59.231+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:59.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:59.231+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:59.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:59.231+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:28:59.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:28:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:28:59.232+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:28:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:59.193+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:59.193+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:59.193+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:28:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:28:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:28:59.193+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:28:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:59.274+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:59.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:59.274+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:59.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:59.274+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:28:59.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:28:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:28:59.275+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:00.539 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:00.255+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:00.540 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:00.255+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:00.540 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:00.255+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:00.540 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:00.256+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:00.540 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:00.240+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:00.540 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:00.240+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:00.540 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:00.240+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:00.540 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:00.240+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:00.541 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:00.225+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:00.541 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:00.226+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:00.541 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:00.226+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:00.541 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:00.226+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:00.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:29:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:29:00.538+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10614 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:29:01.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:01.242+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:01.242+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:01.242+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:01.243+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:01.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:01.205+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:01.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:01.205+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:01.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:01.205+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:01.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:01.206+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:01.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:01.246+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:01.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:01.246+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:01.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:01.246+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:01.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:01.247+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:02.500 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:02.291+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:02.500 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:02.291+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:02.500 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:02.291+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:02.500 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:02.292+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:02.501 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:02.178+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:02.501 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:02.178+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:02.501 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:02.178+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:02.501 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:02.179+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:02.501 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:02.247+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:02.501 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:02.247+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:02.501 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:02.247+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:02.501 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:02.248+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:03.561 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:03.250+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:03.562 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:03.250+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:03.562 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:03.250+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:03.562 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:03.251+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:03.562 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:03.141+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:03.562 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:03.141+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:03.562 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:03.141+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:03.562 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:03.141+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:03.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:03.311+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:03.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:03.311+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:03.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:03.311+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:03.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:03.312+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:04.250+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:04.250+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:04.250+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:04.251+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:04.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:04.111+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:04.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:04.111+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:04.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:04.111+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:04.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:04.112+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:04.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:04.331+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:04.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:04.331+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:04.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:04.331+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:04.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:04.332+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:05.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:05.082+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:05.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:05.082+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:05.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:05.082+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:05.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:05.082+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:05.211+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:05.211+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:05.211+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:05.212+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:05.666 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:29:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:29:05.540+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10709 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:29:05.666 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:05.377+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:05.666 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:05.377+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:05.666 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:05.377+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:05.666 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:05.378+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:06.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:06.234+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:06.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:06.234+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:06.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:06.234+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:06.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:06.234+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:06.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:06.342+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:06.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:06.342+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:06.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:06.342+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:06.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:06.342+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:06.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:06.106+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:06.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:06.106+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:06.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:06.106+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:06.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:06.107+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:07.185+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:07.185+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:07.185+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:07.185+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:07.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:07.148+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:07.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:07.148+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:07.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:07.148+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:07.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:07.148+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:07.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:07.388+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:07.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:07.388+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:07.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:07.388+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:07.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:07.389+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:08.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:08.159+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:08.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:08.159+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:08.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:08.159+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:08.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:08.160+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:08.437 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:08.129+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:08.437 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:08.129+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:08.437 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:08.129+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:08.437 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:08.130+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:08.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:08.436+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:08.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:08.436+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:08.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:08.436+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:08.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:08.437+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:09.155+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:09.155+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:09.155+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:09.155+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:09.394+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:09.394+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:09.394+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:09.394+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:09.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:09.174+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:09.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:09.174+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:09.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:09.174+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:09.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:09.175+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:10.395 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:10.170+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:10.395 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:10.170+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:10.395 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:10.170+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:10.395 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:10.171+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:10.396 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:10.220+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:10.396 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:10.220+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:10.396 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:10.220+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:10.396 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:10.221+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:10.666 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:29:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:29:10.542+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10800 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:29:10.666 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:10.395+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:10.666 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:10.395+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:10.666 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:10.395+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:10.666 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:10.396+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:11.442 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:11.167+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:11.443 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:11.167+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:11.443 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:11.167+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:11.443 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:11.168+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:11.443 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:11.239+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:11.443 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:11.239+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:11.444 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:11.239+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:11.444 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:11.240+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:11.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:11.442+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:11.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:11.442+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:11.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:11.442+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:11.717 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:11.443+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:12.482 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:12.141+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:12.483 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:12.141+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:12.483 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:12.141+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:12.483 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:12.142+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:12.483 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:12.231+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:12.483 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:12.231+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:12.483 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:12.231+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:12.483 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:12.232+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:12.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:12.482+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:12.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:12.482+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:12.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:12.482+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:12.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:12.483+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:13.469 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:13.145+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:13.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:13.145+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:13.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:13.145+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:13.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:13.145+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:13.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:13.228+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:13.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:13.228+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:13.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:13.228+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:13.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:13.229+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:13.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:13.468+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:13.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:13.468+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:13.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:13.468+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:13.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:13.469+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:14.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:14.223+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:14.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:14.223+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:14.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:14.223+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:14.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:14.224+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:14.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:14.118+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:14.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:14.118+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:14.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:14.118+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:14.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:14.118+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:14.789 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:14.473+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:14.789 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:14.473+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:14.789 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:14.473+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:14.789 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:14.474+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:15.440 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:15.096+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:15.440 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:15.096+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:15.440 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:15.096+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:15.441 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:15.097+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:15.441 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:15.273+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:15.441 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:15.273+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:15.441 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:15.273+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:15.441 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:15.274+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:15.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:29:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:29:15.542+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10895 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:29:15.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:15.439+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:15.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:15.439+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:15.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:15.439+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:15.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:15.440+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:16.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:16.121+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:16.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:16.121+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:16.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:16.122+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:16.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:16.122+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:16.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:16.244+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:16.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:16.244+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:16.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:16.244+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:16.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:16.245+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:16.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:16.449+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:16.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:16.449+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:16.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:16.449+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:16.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:16.450+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:17.410 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:17.119+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:17.410 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:17.119+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:17.410 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:17.119+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:17.411 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:17.120+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:17.411 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:17.226+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:17.411 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:17.226+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:17.411 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:17.226+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:17.411 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:17.227+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:17.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:17.409+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:17.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:17.409+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:17.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:17.409+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:17.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:17.410+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:18.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:18.071+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:18.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:18.071+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:18.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:18.071+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:18.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:18.072+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:18.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:18.188+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:18.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:18.188+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:18.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:18.188+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:18.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:18.189+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:18.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:18.360+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:18.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:18.360+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:18.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:18.360+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:18.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:18.361+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:19.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:19.159+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:19.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:19.159+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:19.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:19.159+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:19.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:19.160+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:19.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:19.054+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:19.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:19.054+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:19.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:19.054+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:19.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:19.054+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:19.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:19.379+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:19.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:19.379+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:19.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:19.379+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:19.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:19.379+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:20.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:20.206+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:20.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:20.206+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:20.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:20.206+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:20.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:20.207+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:20.045+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:20.045+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:20.045+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:20.046+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:20.668 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:29:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:29:20.543+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10988 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:29:20.669 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:20.350+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:20.669 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:20.350+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:20.669 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:20.350+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:20.669 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:20.351+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:21.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:21.026+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:21.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:21.026+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:21.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:21.026+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:21.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:21.027+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:21.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:21.170+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:21.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:21.170+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:21.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:21.170+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:21.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:21.171+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:21.676 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:21.391+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:21.676 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:21.391+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:21.676 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:21.391+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:21.676 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:21.392+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:22.023+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:22.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:22.023+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:22.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:22.023+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:22.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:22.024+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:22.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:22.138+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:22.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:22.138+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:22.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:22.138+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:22.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:22.139+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:22.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:22.427+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:22.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:22.427+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:22.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:22.427+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:22.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:22.428+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:23.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:23.013+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:23.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:23.013+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:23.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:23.013+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:23.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:23.014+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:23.120+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:23.120+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:23.120+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:23.121+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:23.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:23.461+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:23.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:23.461+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:23.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:23.461+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:23.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:23.462+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:24.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:23.989+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:24.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:23.989+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:24.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:23.989+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:24.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:23.991+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:24.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:24.142+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:24.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:24.142+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:24.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:24.142+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:24.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:24.143+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:24.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:24.451+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:24.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:24.451+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:24.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:24.451+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:24.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:24.452+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:25.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:24.975+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:25.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:24.975+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:25.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:24.975+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:25.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:24.976+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:25.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:25.128+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:25.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:25.128+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:25.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:25.128+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:25.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:25.128+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:25.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:29:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:29:25.545+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11083 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:29:25.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:25.442+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:25.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:25.442+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:25.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:25.442+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:25.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:25.443+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:26.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:26.019+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:26.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:26.019+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:26.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:26.019+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:26.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:26.020+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:26.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:26.144+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:26.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:26.144+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:26.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:26.144+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:26.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:26.144+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:26.842 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:26.434+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:26.842 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:26.434+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:26.843 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:26.434+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:26.843 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:26.435+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:27.031+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:27.031+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:27.031+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:27.031+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:27.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:27.180+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:27.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:27.180+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:27.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:27.180+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:27.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:27.181+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:27.438+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:27.438+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:27.438+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:27.439+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:28.079+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:28.079+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:28.079+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:28.080+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:28.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:28.206+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:28.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:28.206+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:28.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:28.206+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:28.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:28.206+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:28.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:28.457+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:28.457+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:28.457+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:28.457+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:29.459 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:29.127+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:29.459 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:29.128+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:29.459 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:29.128+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:29.460 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:29.129+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:29.460 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:29.172+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:29.460 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:29.172+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:29.460 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:29.172+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:29.460 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:29.173+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:29.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:29.458+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:29.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:29.458+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:29.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:29.458+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:29.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:29.459+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:30.150 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:30.140+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:30.151 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:30.140+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:30.151 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:30.140+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:30.151 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:30.141+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:30.499 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:30.149+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:30.499 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:30.149+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:30.499 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:30.149+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:30.499 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:30.150+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:30.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:29:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:29:30.546+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11176 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:29:30.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:30.498+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:30.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:30.498+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:30.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:30.498+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:30.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:30.499+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:31.501 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:31.120+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:31.501 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:31.120+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:31.501 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:31.120+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:31.501 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:31.121+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:31.501 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:31.116+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:31.501 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:31.116+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:31.501 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:31.116+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:31.501 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:31.116+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:31.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:31.500+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:31.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:31.500+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:31.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:31.500+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:31.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:31.501+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:32.480 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:32.150+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:32.480 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:32.150+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:32.480 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:32.150+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:32.481 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:32.151+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:32.481 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:32.159+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:32.481 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:32.159+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:32.481 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:32.159+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:32.481 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:32.160+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:32.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:32.479+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:32.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:32.479+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:32.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:32.479+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:32.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:32.480+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:33.458 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:33.164+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:33.458 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:33.164+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:33.458 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:33.164+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:33.458 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:33.165+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:33.458 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:33.167+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:33.459 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:33.167+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:33.459 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:33.167+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:33.459 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:33.168+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:33.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:33.457+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:33.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:33.457+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:33.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:33.457+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:33.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:33.457+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:34.488 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:34.162+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:34.488 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:34.162+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:34.488 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:34.162+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:34.488 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:34.163+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:34.489 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:34.214+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:34.489 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:34.214+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:34.489 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:34.214+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:34.489 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:34.215+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:34.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:34.487+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:34.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:34.487+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:34.487+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:34.488+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:35.478 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:35.188+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:35.478 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:35.188+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:35.478 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:35.188+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:35.478 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:35.189+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:35.478 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:35.118+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:35.478 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:35.118+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:35.478 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:35.118+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:35.478 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:35.119+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:35.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:29:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:29:35.547+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11271 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:29:35.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:35.476+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:35.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:35.476+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:35.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:35.476+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:35.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:35.477+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:36.210 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:36.070+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:36.210 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:36.070+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:36.210 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:36.070+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:36.211 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:36.071+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:36.485 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:36.209+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:36.485 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:36.209+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:36.485 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:36.209+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:36.485 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:36.210+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:36.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:36.484+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:36.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:36.484+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:36.484+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:36.485+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:37.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:37.079+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:37.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:37.079+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:37.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:37.079+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:37.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:37.080+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:37.173+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:37.173+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:37.173+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:37.174+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:37.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:37.467+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:37.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:37.467+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:37.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:37.467+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:37.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:37.468+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:38.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:38.070+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:38.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:38.070+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:38.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:38.070+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:38.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:38.070+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:38.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:38.184+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:38.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:38.184+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:38.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:38.184+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:38.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:38.185+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:38.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:38.496+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:38.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:38.496+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:38.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:38.496+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:38.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:38.497+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:39.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:39.039+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:39.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:39.039+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:39.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:39.039+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:39.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:39.039+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:39.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:39.180+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:39.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:39.180+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:39.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:39.180+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:39.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:39.181+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:39.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:39.456+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:39.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:39.456+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:39.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:39.456+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:39.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:39.457+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:40.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:40.151+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:40.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:40.151+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:40.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:40.151+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:40.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:40.151+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:40.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:40.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:40.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:40.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:40.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:40.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:40.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:40.022+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:40.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:29:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:29:40.549+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11363 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:29:40.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:40.452+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:40.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:40.452+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:40.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:40.452+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:40.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:40.453+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:41.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:41.184+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:41.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:41.184+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:41.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:41.184+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:41.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:41.185+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:41.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:41.020+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:41.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:41.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:41.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:41.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:41.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:41.021+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:41.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:41.496+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:41.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:41.496+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:41.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:41.496+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:41.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:41.497+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:42.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:42.202+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:42.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:42.202+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:42.341 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:42.202+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:42.341 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:42.203+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:42.342 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:41.984+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:42.344 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:41.984+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:42.344 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:41.984+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:42.344 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:41.985+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:42.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:42.506+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:42.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:42.506+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:42.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:42.506+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:42.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:42.507+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:43.219+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:43.219+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:43.219+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:43.220+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:43.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:43.018+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:43.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:43.018+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:43.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:43.018+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:43.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:43.019+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:43.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:43.544+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:43.544+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:43.544+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:43.545+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:44.255+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:44.255+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:44.255+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:44.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:44.256+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:44.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:44.045+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:44.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:44.045+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:44.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:44.045+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:44.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:44.045+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:44.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:44.528+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:44.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:44.528+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:44.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:44.528+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:44.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:44.528+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:45.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:45.072+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:45.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:45.072+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:45.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:45.072+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:45.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:45.072+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:45.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:45.253+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:45.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:45.253+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:45.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:45.253+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:45.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:45.254+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:45.841 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:29:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:29:45.550+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11457 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:29:45.842 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:45.572+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:45.842 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:45.572+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:45.842 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:45.572+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:45.842 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:45.573+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:46.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:46.259+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:46.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:46.259+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:46.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:46.259+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:46.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:46.259+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:46.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:46.073+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:46.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:46.073+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:46.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:46.073+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:46.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:46.074+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:46.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:46.572+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:46.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:46.572+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:46.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:46.572+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:46.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:46.573+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:47.543 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:47.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:47.544 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:47.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:47.544 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:47.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:47.544 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:47.273+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:47.544 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:47.102+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:47.544 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:47.102+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:47.544 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:47.102+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:47.544 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:47.103+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:47.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:47.542+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:47.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:47.542+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:47.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:47.542+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:47.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:47.543+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:48.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:48.322+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:48.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:48.322+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:48.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:48.322+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:48.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:48.323+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:48.559 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:48.133+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:48.560 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:48.133+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:48.560 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:48.133+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:48.560 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:48.134+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:48.560 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:48.558+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:48.560 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:48.558+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:48.560 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:48.558+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:48.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:48.558+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:49.536 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:49.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:49.536 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:49.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:49.536 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:49.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:49.537 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:49.327+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:49.537 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:49.097+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:49.537 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:49.097+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:49.537 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:49.097+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:49.537 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:49.098+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:49.537 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:49.535+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:49.537 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:49.535+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:49.538 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:49.535+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:49.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:49.536+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:50.531 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:50.351+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:50.531 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:50.351+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:50.532 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:50.351+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:50.532 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:50.352+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:50.532 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:50.108+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:50.532 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:50.108+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:50.532 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:50.108+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:50.532 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:50.108+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:50.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:29:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:29:50.552+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11549 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:29:50.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:50.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:50.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:50.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:50.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:50.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:50.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:50.531+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:51.395 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:51.143+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:51.396 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:51.143+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:51.396 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:51.143+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:51.396 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:51.144+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:51.652 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:51.394+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:51.652 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:51.394+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:51.652 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:51.394+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:51.652 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:51.395+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:51.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:51.542+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:51.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:51.542+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:51.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:51.542+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:51.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:51.542+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:52.567 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:52.421+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:52.567 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:52.421+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:52.567 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:52.421+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:52.567 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:52.422+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:52.567 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:52.191+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:52.567 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:52.191+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:52.568 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:52.191+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:52.568 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:52.192+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:52.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:52.565+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:52.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:52.565+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:52.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:52.565+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:52.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:52.566+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:53.428 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:53.151+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:53.428 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:53.151+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:53.428 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:53.151+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:53.429 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:53.152+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:53.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:53.534+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:53.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:53.534+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:53.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:53.534+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:53.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:53.535+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:53.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:53.427+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:53.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:53.427+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:53.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:53.427+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:53.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:53.428+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:54.444 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:54.163+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:54.444 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:54.163+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:54.444 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:54.163+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:54.444 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:54.164+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:54.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:54.442+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:54.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:54.442+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:54.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:54.442+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:54.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:54.443+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:54.511+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:54.511+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:54.511+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:54.511+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:55.555 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:55.411+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:55.555 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:55.411+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:55.555 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:55.411+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:55.555 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:55.412+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:55.555 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:55.200+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:55.555 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:55.200+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:55.555 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:55.200+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:55.555 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:55.201+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:55.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:29:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:29:55.553+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11642 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:29:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:55.558+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:55.558+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:55.558+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:55.558+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:56.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:56.377+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:56.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:56.377+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:56.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:56.377+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:56.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:56.378+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:56.577 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:56.228+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:56.577 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:56.228+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:56.577 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:56.228+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:56.577 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:56.229+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:56.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:56.575+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:56.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:56.575+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:56.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:56.575+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:56.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:56.576+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:57.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:57.545+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:57.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:57.545+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:57.547 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:57.414+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:57.547 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:57.414+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:57.547 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:57.414+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:57.547 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:57.415+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:57.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:57.253+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:57.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:57.253+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:57.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:57.253+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:57.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:57.254+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:57.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:57.545+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:57.546+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:58.507 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:58.440+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:58.508 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:58.440+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:58.508 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:58.440+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:58.508 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:58.441+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:58.508 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:58.210+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:58.508 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:58.210+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:58.508 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:58.210+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:58.508 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:58.210+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:58.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:58.506+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:58.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:58.506+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:58.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:58.506+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:58.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:58.507+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:59.520 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:59.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:59.520 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:59.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:59.521 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:59.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:29:59.521 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:29:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:29:59.455+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:29:59.521 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:59.259+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:59.521 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:59.259+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:59.521 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:59.259+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:29:59.521 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:29:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:29:59.260+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:29:59.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:59.519+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:59.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:59.519+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:59.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:59.519+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:29:59.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:29:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:29:59.520+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:00.478 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:00.275+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:00.478 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:00.275+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:00.478 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:00.275+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:00.478 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:00.276+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:00.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:30:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:30:00.554+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11734 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:30:00.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:00.477+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:00.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:00.477+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:00.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:00.477+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:00.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:00.478+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:00.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:00.490+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:00.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:00.490+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:00.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:00.490+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:00.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:00.490+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:01.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:01.324+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:01.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:01.324+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:01.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:01.324+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:01.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:01.325+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:01.445+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:01.445+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:01.445+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:01.446+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:01.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:01.464+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:01.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:01.464+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:01.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:01.464+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:01.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:01.464+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:02.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:02.469+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:02.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:02.469+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:02.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:02.469+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:02.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:02.470+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:02.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:02.305+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:02.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:02.305+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:02.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:02.305+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:02.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:02.306+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:02.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:02.413+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:02.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:02.413+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:02.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:02.413+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:02.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:02.414+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:03.564 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:03.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:03.564 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:03.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:03.564 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:03.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:03.565 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:03.402+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:03.565 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:03.292+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:03.565 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:03.292+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:03.565 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:03.292+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:03.565 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:03.293+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:03.565 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:03.487+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:03.565 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:03.487+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:03.566 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:03.487+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:03.566 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:03.487+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:04.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:04.301+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:04.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:04.301+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:04.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:04.301+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:04.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:04.302+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:04.534+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:04.534+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:04.534+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:04.535+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:04.438+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:04.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:04.438+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:04.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:04.438+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:04.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:04.439+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:05.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:30:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:30:05.556+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11826 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:30:05.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:05.402+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:05.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:05.402+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:05.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:05.402+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:05.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:05.403+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:05.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:05.336+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:05.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:05.336+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:05.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:05.336+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:05.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:05.337+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:05.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:05.487+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:05.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:05.487+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:05.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:05.487+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:05.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:05.488+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:06.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:06.387+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:06.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:06.387+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:06.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:06.387+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:06.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:06.388+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:06.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:06.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:06.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:06.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:06.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:06.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:06.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:06.360+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:06.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:06.486+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:06.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:06.486+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:06.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:06.486+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:06.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:06.487+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:07.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:07.341+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:07.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:07.341+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:07.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:07.341+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:07.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:07.342+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:07.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:07.352+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:07.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:07.352+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:07.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:07.352+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:07.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:07.353+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:07.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:07.461+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:07.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:07.461+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:07.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:07.461+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:07.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:07.462+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:08.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:08.327+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:08.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:08.327+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:08.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:08.327+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:08.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:08.327+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:08.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:08.315+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:08.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:08.315+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:08.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:08.315+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:08.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:08.316+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:08.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:08.424+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:08.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:08.424+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:08.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:08.424+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:08.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:08.425+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:09.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:09.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:09.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:09.360+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:09.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:09.341+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:09.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:09.341+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:09.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:09.341+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:09.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:09.341+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:09.379+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:09.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:09.379+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:09.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:09.379+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:09.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:09.379+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:10.570 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:30:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:30:10.557+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11918 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:30:10.570 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:10.315+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:10.571 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:10.315+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:10.571 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:10.315+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:10.571 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:10.316+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:10.571 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:10.370+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:10.571 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:10.370+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:10.571 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:10.370+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:10.571 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:10.371+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:10.571 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:10.426+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:10.572 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:10.426+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:10.572 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:10.426+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:10.572 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:10.427+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:11.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:11.407+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:11.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:11.407+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:11.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:11.407+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:11.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:11.408+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:11.334+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:11.334+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:11.334+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:11.335+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:11.387+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:11.387+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:11.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:11.387+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:11.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:11.388+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:12.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:12.372+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:12.372+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:12.372+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:12.373+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:12.438+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:12.438+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:12.438+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:12.439+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:12.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:12.412+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:12.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:12.412+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:12.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:12.412+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:12.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:12.413+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:13.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:13.337+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:13.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:13.337+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:13.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:13.337+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:13.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:13.337+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:13.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:13.417+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:13.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:13.417+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:13.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:13.417+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:13.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:13.418+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:13.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:13.416+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:13.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:13.416+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:13.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:13.416+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:13.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:13.417+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:14.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:14.386+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:14.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:14.386+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:14.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:14.386+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:14.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:14.388+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:14.408+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:14.408+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:14.408+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:14.408+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:14.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:14.406+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:14.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:14.406+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:14.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:14.406+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:14.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:14.406+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:15.652 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:30:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:30:15.558+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12013 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:30:15.653 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:15.390+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:15.653 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:15.390+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:15.653 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:15.390+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:15.653 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:15.391+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:15.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:15.417+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:15.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:15.417+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:15.654 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:15.417+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:15.654 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:15.418+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:15.654 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:15.362+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:15.654 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:15.362+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:15.654 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:15.362+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:15.654 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:15.363+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:16.423+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:16.423+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:16.423+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:16.424+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:16.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:16.338+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:16.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:16.338+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:16.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:16.338+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:16.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:16.338+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:16.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:16.440+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:16.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:16.440+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:16.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:16.440+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:16.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:16.441+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:17.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:17.310+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:17.310+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:17.310+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:17.311+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:17.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:17.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:17.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:17.434+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:17.422+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:17.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:17.422+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:17.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:17.422+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:17.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:17.423+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:18.428+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:18.428+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:18.428+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:18.430+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:18.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:18.294+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:18.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:18.294+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:18.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:18.294+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:18.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:18.295+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:18.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:18.393+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:18.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:18.393+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:18.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:18.393+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:18.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:18.393+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:19.417+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:19.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:19.417+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:19.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:19.417+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:19.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:19.417+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:19.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:19.335+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:19.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:19.335+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:19.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:19.335+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:19.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:19.336+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:19.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:19.373+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:19.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:19.373+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:19.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:19.373+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:19.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:19.374+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:20.561 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:20.386+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:20.561 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:20.386+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:20.561 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:20.386+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:20.561 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:20.387+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:20.562 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:20.330+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:20.562 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:20.330+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:20.562 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:20.330+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:20.562 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:20.330+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:20.562 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:20.384+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:20.562 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:20.384+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:20.562 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:20.384+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:20.562 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:20.385+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:20.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:30:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:30:20.560+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12106 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:30:21.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:30:21.496+0000 7ff1f6bbe700 0 monclient(hunting): authenticate timed out after 300 2024-03-19T03:30:21.497 INFO:teuthology.orchestra.run.smithi059.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-19T03:30:21.680 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:21.371+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:21.680 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:21.371+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:21.680 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:21.371+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:21.680 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:21.371+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:21.680 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:21.424+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:21.680 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:21.424+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:21.681 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:21.424+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:21.681 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:21.424+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:21.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:21.375+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:21.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:21.375+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:21.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:21.375+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:21.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:21.376+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:22.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:22.419+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:22.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:22.419+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:22.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:22.419+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:22.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:22.419+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:22.336+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:22.336+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:22.336+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:22.337+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:22.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:22.407+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:22.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:22.407+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:22.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:22.407+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:22.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:22.408+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:23.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:23.425+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:23.425+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:23.425+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:23.425+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:23.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:23.380+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:23.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:23.380+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:23.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:23.380+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:23.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:23.380+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:23.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:23.451+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:23.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:23.451+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:23.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:23.451+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:23.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:23.451+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:24.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:24.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:24.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:24.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:24.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:24.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:24.360+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:24.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:24.492+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:24.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:24.492+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:24.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:24.492+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:24.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:24.493+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:24.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:24.392+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:24.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:24.392+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:24.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:24.392+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:24.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:24.392+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:25.582 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:30:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:30:25.562+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12202 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:30:25.583 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:25.392+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:25.583 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:25.392+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:25.583 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:25.392+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:25.583 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:25.393+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:25.583 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:25.322+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:25.583 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:25.322+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:25.583 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:25.322+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:25.584 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:25.323+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:25.584 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:25.464+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:25.584 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:25.464+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:25.584 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:25.464+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:25.584 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:25.465+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:26.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:26.318+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:26.318+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:26.318+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:26.319+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:26.425+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:26.425+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:26.425+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:26.426+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:26.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:26.378+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:26.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:26.378+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:26.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:26.378+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:26.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:26.379+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:27.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:27.430+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:27.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:27.430+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:27.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:27.430+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:27.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:27.431+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:27.299+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:27.299+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:27.299+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:27.300+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:27.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:27.418+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:27.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:27.418+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:27.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:27.418+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:27.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:27.419+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:28.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:28.425+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:28.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:28.425+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:28.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:28.425+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:28.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:28.426+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:28.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:28.263+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:28.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:28.263+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:28.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:28.263+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:28.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:28.264+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:28.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:28.430+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:28.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:28.430+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:28.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:28.430+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:28.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:28.430+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:29.240+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:29.240+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:29.240+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:29.240+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:29.419+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:29.419+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:29.419+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:29.419+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:29.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:29.463+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:29.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:29.463+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:29.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:29.463+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:29.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:29.464+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:30.429 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:30.194+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:30.430 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:30.194+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:30.430 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:30.194+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:30.430 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:30.196+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:30.688 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:30:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:30:30.564+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12295 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:30:30.689 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:30.451+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:30.689 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:30.451+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:30.689 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:30.451+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:30.689 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:30.453+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:30.689 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:30.429+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:30.689 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:30.429+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:30.690 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:30.429+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:30.690 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:30.430+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:31.439 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:31.178+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:31.440 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:31.178+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:31.440 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:31.178+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:31.440 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:31.179+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:31.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:31.439+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:31.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:31.439+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:31.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:31.439+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:31.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:31.440+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:31.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:31.446+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:31.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:31.446+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:31.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:31.446+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:31.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:31.446+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:32.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:32.144+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:32.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:32.144+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:32.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:32.144+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:32.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:32.145+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:32.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:32.451+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:32.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:32.451+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:32.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:32.451+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:32.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:32.452+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:32.459+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:32.459+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:32.459+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:32.460+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:33.464 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:33.185+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:33.464 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:33.185+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:33.464 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:33.185+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:33.464 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:33.185+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:33.465 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:33.414+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:33.465 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:33.414+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:33.465 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:33.414+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:33.465 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:33.415+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:33.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:33.463+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:33.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:33.463+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:33.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:33.463+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:33.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:33.464+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:34.509 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:34.217+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:34.510 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:34.217+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:34.510 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:34.217+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:34.510 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:34.218+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:34.510 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:34.422+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:34.510 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:34.422+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:34.510 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:34.422+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:34.510 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:34.423+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:34.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:34.508+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:34.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:34.508+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:34.508+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:34.510+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:35.465 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:35.182+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:35.465 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:35.182+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:35.465 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:35.182+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:35.465 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:35.183+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:35.466 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:35.413+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:35.466 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:35.413+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:35.466 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:35.413+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:35.466 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:35.414+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:35.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:30:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:30:35.564+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12390 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:30:35.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:35.464+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:35.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:35.464+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:35.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:35.464+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:35.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:35.465+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:36.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:36.399+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:36.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:36.399+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:36.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:36.399+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:36.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:36.399+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:36.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:36.232+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:36.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:36.232+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:36.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:36.232+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:36.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:36.233+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:36.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:36.422+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:36.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:36.422+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:36.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:36.422+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:36.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:36.422+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:37.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:37.411+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:37.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:37.411+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:37.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:37.411+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:37.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:37.411+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:37.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:37.215+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:37.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:37.215+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:37.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:37.215+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:37.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:37.215+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:37.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:37.382+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:37.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:37.382+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:37.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:37.382+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:37.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:37.383+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:38.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:38.399+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:38.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:38.399+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:38.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:38.399+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:38.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:38.400+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:38.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:38.220+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:38.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:38.220+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:38.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:38.220+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:38.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:38.221+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:38.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:38.378+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:38.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:38.378+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:38.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:38.378+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:38.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:38.379+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:39.268+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:39.268+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:39.268+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:39.269+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:39.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:39.401+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:39.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:39.401+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:39.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:39.401+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:39.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:39.402+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:39.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:39.363+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:39.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:39.363+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:39.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:39.363+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:39.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:39.364+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:40.319+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:40.319+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:40.319+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:40.320+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:40.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:40.248+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:40.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:40.248+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:40.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:40.248+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:40.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:40.249+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:40.804 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:30:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:30:40.566+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12483 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:30:40.804 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:40.449+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:40.804 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:40.449+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:40.804 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:40.449+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:40.804 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:40.450+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:41.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:41.312+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:41.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:41.312+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:41.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:41.312+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:41.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:41.313+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:41.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:41.432+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:41.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:41.432+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:41.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:41.432+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:41.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:41.433+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:41.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:41.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:41.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:41.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:41.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:41.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:41.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:41.273+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:42.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:42.284+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:42.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:42.284+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:42.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:42.284+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:42.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:42.285+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:42.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:42.449+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:42.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:42.449+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:42.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:42.449+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:42.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:42.451+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:42.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:42.349+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:42.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:42.349+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:42.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:42.349+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:42.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:42.350+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:43.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:43.320+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:43.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:43.320+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:43.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:43.320+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:43.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:43.321+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:43.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:43.429+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:43.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:43.429+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:43.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:43.429+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:43.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:43.429+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:43.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:43.386+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:43.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:43.387+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:43.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:43.387+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:43.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:43.387+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:44.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:44.278+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:44.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:44.278+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:44.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:44.278+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:44.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:44.279+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:44.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:44.432+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:44.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:44.432+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:44.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:44.432+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:44.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:44.432+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:44.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:44.426+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:44.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:44.426+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:44.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:44.426+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:44.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:44.426+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:45.568 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:45.305+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:45.568 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:45.305+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:45.569 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:45.305+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:45.569 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:45.306+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:45.569 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:45.473+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:45.569 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:45.473+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:45.569 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:45.473+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:45.569 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:45.474+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:45.569 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:45.472+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:45.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:45.472+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:45.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:45.472+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:45.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:45.472+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:45.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:30:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:30:45.567+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12579 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:30:46.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:46.282+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:46.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:46.282+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:46.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:46.282+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:46.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:46.283+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:46.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:46.507+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:46.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:46.507+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:46.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:46.507+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:46.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:46.508+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:46.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:46.441+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:46.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:46.441+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:46.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:46.441+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:46.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:46.442+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:47.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:47.433+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:47.433+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:47.433+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:47.433+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:47.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:47.261+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:47.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:47.261+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:47.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:47.261+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:47.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:47.262+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:47.509+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:47.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:47.509+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:47.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:47.509+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:47.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:47.509+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:48.533 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:48.416+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:48.534 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:48.416+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:48.534 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:48.416+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:48.534 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:48.416+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:48.534 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:48.227+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:48.534 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:48.227+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:48.534 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:48.227+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:48.534 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:48.228+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:48.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:48.532+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:48.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:48.532+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:48.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:48.532+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:48.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:48.533+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:49.509 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:49.205+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:49.510 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:49.205+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:49.510 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:49.205+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:49.510 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:49.205+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:49.510 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:49.427+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:49.510 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:49.427+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:49.511 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:49.427+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:49.511 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:49.427+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:49.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:49.508+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:49.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:49.508+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:49.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:49.508+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:49.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:49.509+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:50.421 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:50.172+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:50.421 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:50.172+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:50.421 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:50.172+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:50.421 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:50.173+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:50.692 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:30:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:30:50.568+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12672 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:30:50.693 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:50.420+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:50.693 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:50.420+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:50.693 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:50.420+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:50.693 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:50.421+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:50.693 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:50.525+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:50.693 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:50.525+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:50.693 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:50.525+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:50.693 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:50.526+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:51.413 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:51.138+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:51.413 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:51.138+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:51.413 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:51.138+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:51.414 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:51.139+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:51.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:51.412+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:51.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:51.412+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:51.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:51.412+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:51.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:51.413+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:51.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:51.517+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:51.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:51.517+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:51.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:51.517+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:51.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:51.518+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:52.460 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:52.126+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:52.460 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:52.126+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:52.460 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:52.126+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:52.460 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:52.126+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:52.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:52.459+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:52.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:52.459+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:52.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:52.459+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:52.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:52.459+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:52.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:52.518+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:52.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:52.518+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:52.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:52.518+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:52.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:52.519+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:53.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:53.077+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:53.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:53.077+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:53.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:53.077+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:53.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:53.078+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:53.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:53.477+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:53.477+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:53.477+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:53.478+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:53.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:53.554+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:53.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:53.554+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:53.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:53.554+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:53.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:53.555+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:54.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:54.080+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:54.080+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:54.080+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:54.081+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:54.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:54.504+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:54.504+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:54.504+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:54.505+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:54.515+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:54.515+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:54.515+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:54.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:54.515+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:55.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:55.068+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:55.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:55.068+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:55.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:55.068+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:55.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:55.069+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:55.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:30:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:30:55.570+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12766 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:30:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:55.536+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:55.536+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:55.536+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:55.537+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:55.515+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:55.515+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:55.515+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:55.516+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:56.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:56.045+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:56.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:56.045+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:56.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:56.046+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:56.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:56.046+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:56.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:56.566+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:56.566+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:56.566+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:56.566+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:56.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:56.501+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:56.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:56.501+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:56.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:56.501+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:56.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:56.502+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:57.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:57.034+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:57.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:57.034+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:57.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:57.034+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:57.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:57.035+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:57.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:57.553+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:57.553+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:57.553+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:57.553+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:57.542+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:57.542+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:57.542+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:57.543+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:58.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:58.058+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:58.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:58.058+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:58.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:58.058+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:58.059+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:58.586+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:58.586+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:58.586+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:58.586+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:58.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:58.520+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:58.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:58.520+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:58.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:58.520+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:58.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:58.521+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:59.513 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:59.107+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:59.513 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:59.107+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:59.514 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:59.107+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:30:59.514 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:30:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:30:59.108+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:30:59.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:59.610+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:59.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:59.610+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:59.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:59.610+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:30:59.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:30:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:30:59.611+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:30:59.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:59.512+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:59.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:59.512+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:59.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:59.512+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:30:59.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:30:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:30:59.513+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:00.513 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:00.092+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:00.513 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:00.092+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:00.513 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:00.092+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:00.513 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:00.093+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:00.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:31:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:31:00.571+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12858 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:31:00.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:00.653+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:00.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:00.653+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:00.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:00.653+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:00.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:00.654+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:00.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:00.511+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:00.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:00.511+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:00.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:00.511+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:00.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:00.512+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:01.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:01.048+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:01.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:01.048+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:01.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:01.048+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:01.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:01.048+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:01.639 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:01.515+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:01.639 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:01.515+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:01.640 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:01.515+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:01.640 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:01.515+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:02.023 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:01.638+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:02.023 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:01.638+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:02.023 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:01.638+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:02.023 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:01.639+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:02.308 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:02.022+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:02.308 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:02.022+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:02.308 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:02.022+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:02.308 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:02.023+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:02.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:02.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:02.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:02.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:02.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:02.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:02.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:02.531+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:03.013 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:02.687+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:03.013 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:02.687+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:03.013 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:02.687+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:03.014 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:02.689+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:03.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:03.012+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:03.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:03.012+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:03.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:03.012+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:03.012+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:03.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:03.498+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:03.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:03.498+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:03.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:03.498+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:03.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:03.499+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:03.703+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:03.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:03.703+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:03.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:03.703+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:03.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:03.704+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:04.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:03.981+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:04.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:03.981+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:04.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:03.981+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:04.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:03.982+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:04.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:04.697+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:04.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:04.697+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:04.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:04.697+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:04.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:04.698+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:04.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:04.505+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:04.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:04.505+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:04.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:04.505+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:04.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:04.506+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:05.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:05.031+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:05.031+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:05.031+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:05.032+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:05.696 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:31:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:31:05.573+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12952 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:31:05.696 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:05.469+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:05.697 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:05.469+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:05.697 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:05.469+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:05.697 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:05.470+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:06.030 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:05.699+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:06.030 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:05.699+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:06.030 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:05.699+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:06.030 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:05.700+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:06.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:06.029+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:06.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:06.029+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:06.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:06.029+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:06.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:06.030+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:06.749 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:06.468+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:06.749 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:06.468+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:06.749 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:06.468+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:06.749 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:06.468+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:07.072 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:06.748+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:07.073 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:06.748+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:07.073 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:06.748+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:07.073 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:06.749+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:07.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:07.071+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:07.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:07.071+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:07.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:07.071+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:07.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:07.072+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:07.712 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:07.460+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:07.712 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:07.460+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:07.712 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:07.460+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:07.712 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:07.461+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:08.025 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:07.711+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:08.025 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:07.711+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:08.025 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:07.711+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:08.025 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:07.712+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:08.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:08.024+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:08.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:08.024+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:08.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:08.024+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:08.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:08.025+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:08.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:08.689+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:08.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:08.689+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:08.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:08.689+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:08.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:08.690+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:08.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:08.441+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:08.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:08.441+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:08.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:08.441+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:08.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:08.441+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:09.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:09.057+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:09.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:09.057+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:09.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:09.057+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:09.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:09.058+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:09.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:09.703+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:09.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:09.703+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:09.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:09.703+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:09.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:09.704+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:09.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:09.491+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:09.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:09.491+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:09.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:09.491+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:09.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:09.492+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:10.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:10.030+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:10.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:10.030+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:10.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:10.030+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:10.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:10.030+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:10.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:31:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:31:10.574+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13044 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:31:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:10.745+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:10.745+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:10.745+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:10.747+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:10.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:10.497+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:10.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:10.497+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:10.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:10.497+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:10.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:10.498+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:11.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:11.003+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:11.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:11.003+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:11.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:11.003+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:11.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:11.004+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:11.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:11.753+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:11.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:11.753+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:11.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:11.753+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:11.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:11.753+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:11.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:11.522+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:11.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:11.522+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:11.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:11.522+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:11.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:11.523+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:12.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:12.001+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:12.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:12.001+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:12.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:12.001+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:12.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:12.002+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:12.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:12.774+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:12.774+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:12.774+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:12.775+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:12.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:12.545+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:12.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:12.545+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:12.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:12.545+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:12.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:12.546+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:13.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:12.994+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:13.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:12.994+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:13.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:12.994+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:13.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:12.995+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:13.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:13.776+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:13.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:13.776+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:13.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:13.776+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:13.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:13.777+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:13.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:13.579+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:13.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:13.579+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:13.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:13.579+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:13.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:13.580+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:14.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:14.005+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:14.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:14.005+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:14.005+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:14.006+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:14.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:14.777+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:14.777+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:14.777+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:14.778+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:14.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:14.564+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:14.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:14.564+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:14.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:14.564+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:14.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:14.565+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:15.109 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:15.017+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:15.109 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:15.017+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:15.109 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:15.017+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:15.109 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:15.018+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:15.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:31:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:31:15.576+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13138 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:31:15.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:15.583+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:15.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:15.583+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:15.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:15.583+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:15.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:15.584+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:15.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:15.746+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:15.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:15.746+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:15.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:15.746+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:15.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:15.747+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:16.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:16.048+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:16.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:16.048+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:16.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:16.048+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:16.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:16.049+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:16.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:16.571+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:16.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:16.571+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:16.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:16.571+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:16.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:16.572+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:16.720+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:16.720+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:16.720+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:16.721+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:17.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:17.052+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:17.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:17.052+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:17.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:17.052+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:17.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:17.053+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:18.064 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:17.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:18.064 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:17.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:18.064 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:17.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:18.064 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:17.745+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:18.064 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:17.618+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:18.064 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:17.618+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:18.065 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:17.618+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:18.065 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:17.619+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:18.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:18.063+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:18.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:18.063+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:18.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:18.063+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:18.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:18.063+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:19.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:18.594+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:19.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:18.594+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:19.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:18.594+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:19.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:18.594+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:18.766+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:18.766+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:18.766+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:19.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:18.767+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:19.093+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:19.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:19.093+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:19.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:19.093+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:19.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:19.095+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:20.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:19.618+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:20.407 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:19.618+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:20.407 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:19.618+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:20.408 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:19.619+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:20.408 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:19.735+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:20.408 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:19.735+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:20.408 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:19.735+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:20.408 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:19.736+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:20.578 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:20.117+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:20.578 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:20.117+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:20.578 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:20.117+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:20.578 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:20.118+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:20.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:31:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:31:20.577+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13230 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:31:20.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:20.653+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:20.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:20.653+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:20.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:20.653+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:20.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:20.653+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:20.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:20.712+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:20.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:20.712+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:20.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:20.712+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:20.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:20.713+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:21.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:21.110+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:21.110+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:21.110+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:21.111+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:22.066 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:21.670+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:22.066 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:21.670+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:22.067 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:21.670+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:22.067 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:21.671+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:22.067 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:21.645+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:22.067 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:21.645+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:22.067 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:21.645+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:22.067 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:21.645+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:22.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:22.065+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:22.065+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:22.065+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:22.066+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:23.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:22.690+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:22.690+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:22.690+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:22.691+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:22.652+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:22.653+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:22.653+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:22.653+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:23.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:23.086+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:23.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:23.086+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:23.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:23.086+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:23.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:23.087+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:23.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:23.717+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:23.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:23.717+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:23.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:23.717+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:23.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:23.717+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:23.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:23.614+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:23.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:23.614+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:23.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:23.614+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:23.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:23.614+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:24.095+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:24.095+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:24.095+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:24.096+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:25.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:24.639+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:25.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:24.639+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:25.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:24.639+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:25.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:24.639+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:24.767+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:24.767+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:24.768+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:24.769+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:25.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:25.141+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:25.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:25.141+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:25.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:25.141+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:25.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:25.142+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:25.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:31:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:31:25.578+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13324 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:31:25.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:25.814+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:25.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:25.814+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:25.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:25.814+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:25.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:25.815+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:25.617+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:25.617+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:25.617+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:25.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:25.619+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:26.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:26.178+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:26.178+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:26.178+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:26.179+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:26.885 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:26.843+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:26.885 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:26.843+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:26.885 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:26.843+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:26.885 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:26.844+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:26.885 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:26.609+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:26.885 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:26.609+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:26.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:26.609+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:26.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:26.610+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:27.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:27.160+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:27.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:27.160+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:27.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:27.160+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:27.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:27.161+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:27.822+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:27.822+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:27.822+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:27.822+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:27.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:27.575+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:27.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:27.575+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:27.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:27.575+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:27.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:27.576+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:28.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:28.118+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:28.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:28.118+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:28.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:28.118+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:28.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:28.119+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:28.873 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:28.596+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:28.873 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:28.596+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:28.873 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:28.596+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:28.873 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:28.597+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:29.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:29.071+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:29.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:29.071+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:29.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:29.071+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:29.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:29.071+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:29.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:28.872+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:29.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:28.872+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:29.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:28.872+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:29.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:28.873+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:30.076 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:29.589+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:30.076 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:29.589+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:30.076 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:29.590+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:30.076 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:29.591+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:30.076 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:29.836+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:30.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:29.836+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:30.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:29.836+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:30.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:29.837+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:30.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:30.075+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:30.075+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:30.075+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:30.076+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:30.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:31:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:31:30.580+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13416 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:31:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:30.596+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:30.596+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:30.596+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:30.597+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:31.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:30.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:31.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:30.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:31.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:30.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:31.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:30.861+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:31.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:31.096+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:31.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:31.096+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:31.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:31.096+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:31.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:31.097+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:31.839+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:31.839+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:31.839+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:31.840+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:31.602+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:31.602+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:31.602+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:31.602+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:32.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:32.132+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:32.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:32.132+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:32.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:32.132+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:32.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:32.133+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:33.030 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:32.601+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:33.030 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:32.601+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:33.030 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:32.601+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:33.030 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:32.601+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:33.030 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:32.797+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:33.031 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:32.797+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:33.031 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:32.797+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:33.031 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:32.797+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:33.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:33.171+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:33.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:33.171+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:33.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:33.171+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:33.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:33.171+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:33.819 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:33.817+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:33.819 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:33.817+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:33.819 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:33.817+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:33.819 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:33.818+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:33.819 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:33.642+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:33.819 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:33.642+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:33.819 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:33.642+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:33.820 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:33.642+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:34.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:34.204+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:34.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:34.204+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:34.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:34.204+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:34.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:34.205+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:35.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:34.671+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:35.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:34.671+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:35.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:34.671+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:35.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:34.672+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:35.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:34.823+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:35.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:34.823+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:35.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:34.823+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:35.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:34.823+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:35.582 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:35.253+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:35.582 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:35.253+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:35.582 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:35.253+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:35.582 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:35.254+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:35.839 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:31:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:31:35.582+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13510 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:31:35.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:35.682+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:35.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:35.682+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:35.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:35.682+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:35.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:35.683+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:35.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:35.822+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:35.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:35.822+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:35.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:35.822+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:35.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:35.822+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:36.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:36.226+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:36.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:36.226+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:36.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:36.226+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:36.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:36.227+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:37.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:36.685+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:37.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:36.685+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:37.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:36.685+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:37.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:36.686+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:36.789+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:36.789+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:36.789+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:36.789+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:37.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:37.269+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:37.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:37.269+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:37.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:37.269+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:37.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:37.269+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:38.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:37.710+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:38.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:37.710+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:38.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:37.710+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:38.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:37.710+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:38.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:37.819+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:38.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:37.819+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:38.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:37.819+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:38.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:37.820+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:38.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:38.224+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:38.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:38.224+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:38.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:38.224+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:38.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:38.225+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:39.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:38.714+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:38.714+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:38.714+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:38.714+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:38.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:38.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:38.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:38.853+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:39.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:39.233+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:39.233+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:39.233+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:39.234+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:40.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:39.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:39.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:39.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:39.859+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:39.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:39.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:39.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:39.763+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:40.584 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:40.282+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:40.584 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:40.282+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:40.584 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:40.282+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:40.585 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:40.283+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:40.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:31:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:31:40.583+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13602 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:31:40.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:40.827+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:40.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:40.827+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:40.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:40.827+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:40.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:40.828+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:40.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:40.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:40.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:40.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:40.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:40.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:40.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:40.770+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:41.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:41.295+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:41.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:41.295+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:41.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:41.295+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:41.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:41.296+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:42.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:41.799+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:42.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:41.799+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:42.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:41.799+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:42.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:41.800+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:42.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:41.842+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:42.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:41.842+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:42.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:41.842+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:42.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:41.843+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:42.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:42.333+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:42.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:42.333+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:42.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:42.333+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:42.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:42.333+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:43.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:42.832+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:43.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:42.832+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:43.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:42.832+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:43.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:42.832+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:43.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:42.865+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:43.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:42.865+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:43.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:42.865+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:43.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:42.865+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:43.810 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:43.362+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:43.811 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:43.362+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:43.811 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:43.362+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:43.811 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:43.362+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:44.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:43.843+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:44.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:43.843+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:44.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:43.843+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:44.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:43.843+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:44.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:43.810+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:44.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:43.810+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:44.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:43.810+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:44.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:43.811+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:44.812 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:44.410+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:44.813 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:44.410+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:44.813 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:44.410+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:44.813 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:44.411+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:45.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:44.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:45.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:44.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:45.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:44.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:45.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:44.859+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:45.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:44.812+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:45.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:44.812+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:45.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:44.812+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:45.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:44.813+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:45.586 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:45.458+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:45.586 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:45.458+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:45.586 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:45.458+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:45.586 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:45.459+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:45.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:31:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:31:45.586+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13696 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:31:46.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:45.861+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:46.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:45.861+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:46.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:45.861+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:46.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:45.862+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:46.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:45.847+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:46.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:45.847+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:46.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:45.847+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:46.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:45.848+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:46.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:46.415+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:46.415+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:46.415+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:46.415+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:47.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:46.888+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:47.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:46.888+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:47.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:46.888+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:47.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:46.889+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:47.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:46.849+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:47.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:46.849+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:47.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:46.849+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:47.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:46.850+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:47.811 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:47.453+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:47.811 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:47.453+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:47.811 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:47.453+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:47.811 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:47.454+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:48.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:47.919+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:48.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:47.919+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:48.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:47.919+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:48.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:47.920+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:47.811+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:47.811+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:47.811+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:47.811+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:48.793 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:48.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:48.793 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:48.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:48.793 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:48.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:48.793 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:48.434+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:49.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:48.896+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:49.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:48.896+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:49.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:48.896+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:49.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:48.896+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:49.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:48.793+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:49.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:48.793+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:49.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:48.793+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:49.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:48.794+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:49.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:49.477+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:49.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:49.477+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:49.477+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:49.478+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:49.917+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:49.917+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:49.917+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:49.918+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:49.788+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:49.788+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:49.788+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:49.788+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:50.825 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:31:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:31:50.587+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13789 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:31:50.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:50.497+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:50.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:50.497+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:50.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:50.497+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:50.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:50.498+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:51.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:50.959+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:51.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:50.959+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:51.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:50.959+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:51.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:50.960+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:50.825+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:50.825+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:50.825+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:50.825+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:51.814 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:51.529+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:51.815 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:51.529+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:51.815 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:51.529+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:51.815 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:51.530+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:52.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:51.999+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:52.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:51.999+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:51.999+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:52.000+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:51.813+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:51.813+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:51.813+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:51.814+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:52.776 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:52.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:52.777 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:52.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:52.777 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:52.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:52.777 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:52.489+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:53.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:52.982+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:53.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:52.982+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:53.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:52.982+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:53.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:52.983+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:52.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:52.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:52.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:52.776+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:53.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:53.489+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:53.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:53.489+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:53.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:53.489+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:53.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:53.489+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:54.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:53.992+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:54.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:53.992+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:54.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:53.992+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:54.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:53.993+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:54.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:53.822+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:54.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:53.822+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:54.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:53.822+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:54.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:53.823+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:54.526+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:54.526+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:54.526+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:54.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:54.527+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:55.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:54.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:55.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:54.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:55.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:54.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:55.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:54.958+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:55.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:54.851+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:55.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:54.851+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:55.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:54.851+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:55.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:54.851+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:55.821 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:31:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:31:55.588+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13883 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:31:55.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:55.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:55.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:55.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:55.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:55.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:55.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:55.511+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:56.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:55.940+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:56.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:55.940+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:56.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:55.940+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:56.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:55.942+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:56.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:55.820+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:56.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:55.820+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:56.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:55.820+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:56.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:55.821+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:56.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:56.492+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:56.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:56.492+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:56.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:56.492+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:56.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:56.493+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:57.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:56.989+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:57.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:56.989+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:57.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:56.989+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:57.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:56.990+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:56.859+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:56.859+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:56.859+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:56.860+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:57.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:57.540+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:57.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:57.540+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:57.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:57.540+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:57.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:57.540+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:58.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:58.003+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:58.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:58.003+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:58.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:58.003+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:58.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:58.004+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:57.823+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:57.823+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:57.823+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:57.823+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:58.804 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:58.520+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:58.804 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:58.520+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:58.804 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:58.520+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:58.805 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:58.521+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:59.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:59.050+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:59.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:59.050+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:59.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:59.050+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:31:59.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:31:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:31:59.051+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:31:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:58.803+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:58.803+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:58.803+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:31:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:58.804+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:31:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:59.480+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:59.480+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:59.480+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:31:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:31:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:31:59.481+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:00.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:00.022+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:00.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:00.022+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:00.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:00.022+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:00.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:00.023+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:59.828+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:59.828+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:59.828+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:31:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:31:59.829+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:00.805 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:32:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:32:00.589+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13976 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:32:00.805 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:00.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:00.805 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:00.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:00.805 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:00.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:00.805 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:00.489+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:01.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:00.990+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:01.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:00.990+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:00.990+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:00.991+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:00.804+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:00.804+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:00.804+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:00.805+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:01.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:01.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:01.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:01.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:01.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:01.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:01.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:01.489+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:02.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:01.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:01.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:01.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:01.852+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:02.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:02.019+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:02.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:02.019+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:02.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:02.019+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:02.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:02.020+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:02.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:02.485+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:02.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:02.485+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:02.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:02.485+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:02.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:02.486+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:02.884+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:02.884+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:02.884+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:02.884+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:03.015+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:03.015+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:03.015+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:03.016+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:03.748 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:03.496+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:03.748 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:03.496+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:03.748 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:03.496+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:03.748 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:03.497+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:04.022 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:03.844+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:04.023 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:03.844+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:04.023 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:03.844+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:04.023 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:03.844+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:04.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:04.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:04.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:04.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:04.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:04.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:04.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:04.022+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:04.828 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:04.451+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:04.829 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:04.451+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:04.829 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:04.451+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:04.829 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:04.452+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:05.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:04.827+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:04.827+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:04.827+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:04.828+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:05.000+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:05.000+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:05.000+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:05.000+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:05.715 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:32:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:32:05.591+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14070 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:32:05.715 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:05.447+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:05.715 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:05.447+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:05.716 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:05.447+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:05.716 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:05.448+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:05.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:05.814+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:05.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:05.814+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:05.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:05.814+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:05.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:05.815+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:06.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:05.970+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:06.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:05.970+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:06.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:05.970+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:06.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:05.971+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:06.796 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:06.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:06.796 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:06.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:06.796 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:06.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:06.796 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:06.402+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:07.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:07.016+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:07.016+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:07.016+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:07.016+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:06.795+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:06.795+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:06.795+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:06.796+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:07.833 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:07.414+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:07.834 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:07.414+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:07.834 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:07.414+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:07.834 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:07.414+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:08.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:07.988+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:08.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:07.988+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:07.988+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:07.988+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:07.832+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:07.832+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:07.832+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:07.833+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:08.809 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:08.434+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:08.809 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:08.434+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:08.809 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:08.434+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:08.809 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:08.435+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:09.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:09.011+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:09.011+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:09.011+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:09.012+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:08.808+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:08.808+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:08.808+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:08.809+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:09.426+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:09.426+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:09.426+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:09.427+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:10.153 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:10.062+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:10.153 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:10.062+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:10.153 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:10.062+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:10.153 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:10.062+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:10.153 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:09.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:10.153 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:09.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:10.153 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:09.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:10.153 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:09.858+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:10.717 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:32:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:32:10.592+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14164 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:32:10.718 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:10.435+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:10.718 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:10.435+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:10.718 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:10.435+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:10.718 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:10.436+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:11.018 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:10.855+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:11.018 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:10.855+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:11.018 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:10.855+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:11.018 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:10.856+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:11.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:11.017+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:11.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:11.017+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:11.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:11.017+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:11.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:11.018+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:11.685 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:11.428+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:11.686 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:11.428+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:11.686 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:11.428+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:11.686 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:11.429+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:12.012 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:11.850+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:12.012 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:11.850+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:12.012 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:11.850+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:12.012 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:11.851+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:12.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:12.010+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:12.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:12.010+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:12.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:12.010+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:12.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:12.011+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:12.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:12.388+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:12.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:12.388+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:12.388+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:12.389+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:13.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:12.980+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:13.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:12.980+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:13.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:12.980+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:13.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:12.981+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:13.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:12.853+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:13.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:12.853+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:13.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:12.853+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:13.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:12.854+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:13.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:13.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:13.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:13.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:13.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:13.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:13.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:13.397+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:14.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:14.024+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:14.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:14.024+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:14.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:14.024+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:14.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:14.025+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:14.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:13.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:14.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:13.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:14.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:13.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:14.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:13.857+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:14.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:14.375+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:14.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:14.375+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:14.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:14.375+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:14.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:14.376+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:15.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:15.051+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:15.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:15.051+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:15.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:15.051+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:15.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:15.051+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:15.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:14.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:15.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:14.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:15.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:14.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:15.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:14.853+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:15.718 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:32:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:32:15.594+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14254 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:32:15.718 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:15.377+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:15.718 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:15.377+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:15.718 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:15.377+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:15.718 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:15.378+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:16.036 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:15.884+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:16.036 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:15.884+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:16.036 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:15.884+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:16.036 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:15.885+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:16.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:16.058+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:16.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:16.058+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:16.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:16.058+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:16.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:16.059+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:16.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:16.376+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:16.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:16.376+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:16.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:16.376+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:16.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:16.377+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:17.332 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:17.018+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:17.333 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:17.018+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:17.333 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:17.018+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:17.333 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:17.018+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:17.333 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:16.926+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:17.333 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:16.926+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:17.333 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:16.926+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:17.333 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:16.927+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:17.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:17.331+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:17.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:17.331+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:17.331+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:17.332+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:17.962+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:18.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:17.962+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:18.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:17.962+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:18.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:17.963+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:18.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:17.973+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:18.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:17.973+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:18.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:17.973+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:18.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:17.973+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:18.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:18.353+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:18.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:18.353+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:18.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:18.353+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:18.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:18.354+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:19.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:19.007+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:19.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:19.007+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:19.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:19.007+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:19.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:19.008+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:19.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:18.993+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:19.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:18.993+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:19.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:18.993+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:19.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:18.993+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:19.397+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:19.397+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:19.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:19.397+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:19.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:19.398+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:20.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:19.988+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:20.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:19.988+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:20.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:19.988+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:20.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:19.988+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:20.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:20.011+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:20.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:20.011+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:20.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:20.011+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:20.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:20.012+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:20.719 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:32:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:32:20.595+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14340 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:32:20.720 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:20.387+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:20.720 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:20.387+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:20.720 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:20.387+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:20.720 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:20.388+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:21.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:20.963+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:21.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:20.963+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:21.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:20.963+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:21.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:20.964+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:21.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:20.967+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:21.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:20.967+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:21.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:20.967+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:21.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:20.968+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:21.686 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:21.374+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:21.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:21.374+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:21.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:21.374+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:21.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:21.375+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:22.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:21.976+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:22.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:21.976+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:22.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:21.976+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:22.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:21.977+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:22.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:21.956+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:22.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:21.956+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:22.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:21.956+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:22.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:21.957+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:22.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:22.352+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:22.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:22.352+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:22.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:22.352+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:22.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:22.353+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:23.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:22.944+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:22.944+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:22.944+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:22.944+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:23.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:22.956+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:23.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:22.956+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:23.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:22.956+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:23.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:22.956+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:23.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:23.345+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:23.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:23.345+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:23.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:23.345+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:23.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:23.346+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:24.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:23.907+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:24.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:23.907+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:24.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:23.907+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:24.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:23.907+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:24.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:23.988+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:24.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:23.988+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:24.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:23.988+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:24.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:23.988+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:24.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:24.336+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:24.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:24.336+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:24.336+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:24.336+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:25.252 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:24.990+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:25.252 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:24.990+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:25.252 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:24.990+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:25.252 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:24.990+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:25.253 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:24.874+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:25.253 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:24.874+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:25.253 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:24.874+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:25.253 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:24.875+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:25.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:25.308+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:25.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:25.308+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:25.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:25.308+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:25.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:25.308+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:25.902 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:32:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:32:25.597+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14429 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:32:26.291 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:25.901+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:26.291 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:25.901+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:26.291 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:25.901+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:26.291 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:25.902+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:26.292 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:25.985+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:26.292 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:25.985+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:26.292 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:25.985+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:26.292 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:25.986+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:26.290+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:26.290+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:26.290+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:26.291+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:27.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:27.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:27.294 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:27.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:27.294 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:27.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:27.294 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:27.034+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:27.294 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:26.912+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:27.294 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:26.912+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:27.294 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:26.913+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:27.294 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:26.913+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:27.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:27.292+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:27.292+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:27.292+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:27.293+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:28.325 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:27.907+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:28.325 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:27.907+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:28.325 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:27.907+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:28.325 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:27.908+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:28.325 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:28.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:28.325 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:28.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:28.325 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:28.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:28.326 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:28.032+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:28.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:28.333+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:28.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:28.333+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:28.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:28.333+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:28.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:28.334+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:29.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:29.028+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:29.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:29.028+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:29.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:29.028+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:29.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:29.029+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:29.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:28.954+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:29.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:28.954+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:29.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:28.954+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:29.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:28.954+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:29.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:29.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:29.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:29.362+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:30.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:29.910+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:30.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:29.910+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:30.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:29.910+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:30.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:29.912+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:30.043+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:30.043+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:30.043+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:30.044+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:30.724 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:32:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:32:30.598+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14515 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:32:30.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:30.399+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:30.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:30.399+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:30.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:30.399+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:30.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:30.400+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:31.053 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:30.954+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:31.053 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:30.954+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:31.053 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:30.954+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:31.053 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:30.955+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:31.055 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:31.052+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:31.055 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:31.052+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:31.055 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:31.052+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:31.055 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:31.053+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:31.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:31.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:31.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:31.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:31.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:31.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:31.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:31.434+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:32.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:31.950+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:32.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:31.950+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:32.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:31.950+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:32.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:31.951+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:32.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:32.067+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:32.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:32.067+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:32.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:32.067+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:32.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:32.067+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:32.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:32.472+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:32.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:32.472+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:32.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:32.472+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:32.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:32.474+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:33.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:32.958+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:33.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:32.958+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:33.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:32.958+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:33.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:32.959+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:33.083+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:33.083+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:33.083+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:33.084+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:33.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:33.441+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:33.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:33.441+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:33.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:33.441+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:33.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:33.442+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:34.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:33.982+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:34.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:33.982+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:34.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:33.982+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:34.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:33.983+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:34.052+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:34.052+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:34.052+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:34.053+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:34.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:34.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:34.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:34.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:34.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:34.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:34.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:34.454+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:35.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:34.945+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:35.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:34.945+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:35.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:34.945+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:35.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:34.946+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:35.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:35.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:35.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:35.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:35.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:35.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:35.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:35.052+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:35.728 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:32:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:32:35.600+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14605 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:32:35.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:35.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:35.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:35.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:35.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:35.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:35.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:35.406+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:36.024 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:35.923+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:36.024 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:35.923+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:36.024 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:35.923+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:36.024 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:35.924+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:36.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:36.022+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:36.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:36.022+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:36.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:36.022+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:36.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:36.023+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:36.408+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:36.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:36.408+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:36.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:36.408+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:36.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:36.409+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:37.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:36.984+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:37.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:36.984+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:37.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:36.984+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:37.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:36.985+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:37.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:36.939+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:37.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:36.939+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:37.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:36.939+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:37.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:36.940+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:37.410+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:37.410+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:37.410+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:37.411+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:38.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:37.989+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:38.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:37.989+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:38.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:37.989+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:38.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:37.990+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:38.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:38.005+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:38.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:38.005+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:38.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:38.005+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:38.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:38.005+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:38.842 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:38.437+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:38.842 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:38.437+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:38.842 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:38.437+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:38.842 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:38.438+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:39.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:38.961+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:39.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:38.961+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:39.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:38.961+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:39.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:38.962+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:39.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:39.022+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:39.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:39.022+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:39.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:39.022+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:39.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:39.023+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:39.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:39.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:39.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:39.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:39.450+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:40.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:39.932+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:40.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:39.932+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:40.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:39.932+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:40.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:39.932+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:40.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:40.005+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:40.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:40.005+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:40.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:40.005+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:40.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:40.006+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:40.728 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:32:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:32:40.602+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14691 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:32:40.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:40.402+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:40.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:40.402+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:40.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:40.402+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:40.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:40.403+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:40.996 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:40.894+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:40.996 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:40.894+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:40.996 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:40.894+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:40.997 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:40.895+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:41.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:40.995+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:41.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:40.995+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:41.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:40.995+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:41.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:40.996+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:41.688 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:41.430+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:41.688 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:41.430+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:41.688 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:41.430+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:41.688 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:41.431+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:42.027 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:41.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:42.028 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:41.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:42.028 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:41.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:42.028 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:41.930+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:42.026+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:42.026+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:42.026+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:42.028+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:42.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:42.448+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:42.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:42.448+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:42.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:42.448+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:42.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:42.449+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:42.956+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:42.956+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:42.956+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:42.957+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:43.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:42.980+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:43.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:42.980+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:43.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:42.980+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:43.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:42.980+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:43.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:43.461+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:43.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:43.461+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:43.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:43.461+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:43.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:43.462+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:43.979+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:44.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:43.979+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:44.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:43.979+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:44.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:43.980+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:43.934+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:43.934+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:43.934+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:43.934+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:44.462+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:44.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:44.462+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:44.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:44.462+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:44.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:44.464+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:44.935+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:45.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:44.935+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:45.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:44.935+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:45.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:44.935+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:45.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:44.967+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:45.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:44.967+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:45.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:44.967+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:45.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:44.967+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:45.727 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:32:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:32:45.603+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14780 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:32:45.727 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:45.464+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:45.727 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:45.464+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:45.727 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:45.464+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:45.727 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:45.464+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:46.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:45.922+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:46.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:45.922+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:46.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:45.922+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:46.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:45.923+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:46.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:45.972+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:46.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:45.972+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:46.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:45.972+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:46.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:45.973+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:46.757 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:46.456+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:46.757 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:46.456+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:46.757 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:46.456+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:46.757 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:46.457+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:47.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:46.983+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:47.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:46.983+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:47.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:46.983+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:47.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:46.983+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:46.878+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:46.878+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:46.878+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:46.879+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:47.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:47.434+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:47.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:47.434+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:47.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:47.434+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:47.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:47.435+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:48.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:47.941+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:48.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:47.941+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:48.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:47.941+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:48.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:47.941+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:47.836+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:47.836+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:47.836+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:47.837+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:48.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:48.429+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:48.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:48.429+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:48.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:48.429+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:48.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:48.430+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:49.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:48.911+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:49.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:48.911+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:49.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:48.911+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:49.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:48.911+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:49.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:48.849+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:49.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:48.849+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:49.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:48.849+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:49.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:48.850+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:49.391+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:49.391+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:49.391+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:49.392+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:49.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:49.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:49.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:49.867+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:49.814+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:49.814+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:49.814+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:49.815+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:50.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:50.379+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:50.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:50.379+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:50.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:50.379+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:50.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:50.379+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:50.856 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:32:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:32:50.605+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14866 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:32:50.856 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:50.854+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:50.856 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:50.854+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:50.857 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:50.854+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:50.857 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:50.855+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:51.336 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:50.860+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:51.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:50.860+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:51.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:50.860+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:51.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:50.861+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:51.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:51.335+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:51.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:51.335+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:51.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:51.335+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:51.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:51.336+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:51.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:51.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:51.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:51.857+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:51.867+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:51.867+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:51.868+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:51.868+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:52.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:52.289+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:52.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:52.289+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:52.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:52.289+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:52.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:52.289+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:53.243 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:52.853+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:53.243 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:52.853+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:53.244 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:52.853+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:53.244 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:52.854+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:53.244 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:52.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:53.244 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:52.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:53.244 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:52.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:53.244 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:52.853+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:53.574 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:53.242+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:53.574 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:53.242+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:53.574 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:53.242+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:53.574 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:53.243+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:53.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:53.811+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:53.811+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:53.811+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:53.812+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:54.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:53.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:54.248 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:53.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:54.248 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:53.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:54.248 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:53.848+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:54.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:54.246+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:54.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:54.246+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:54.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:54.246+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:54.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:54.247+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:55.217 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:54.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:55.217 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:54.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:55.217 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:54.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:55.217 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:54.859+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:55.218 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:54.881+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:55.218 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:54.882+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:55.218 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:54.882+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:55.218 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:54.882+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:55.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:55.216+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:55.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:55.216+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:55.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:55.216+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:55.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:55.217+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:55.858 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:32:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:32:55.607+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14956 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:32:56.242 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:55.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:56.242 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:55.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:56.242 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:55.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:56.242 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:55.859+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:56.242 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:55.880+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:56.243 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:55.880+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:56.243 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:55.880+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:56.243 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:55.881+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:56.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:56.241+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:56.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:56.241+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:56.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:56.241+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:56.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:56.241+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:57.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:56.823+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:57.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:56.823+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:57.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:56.823+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:57.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:56.824+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:57.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:56.869+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:57.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:56.869+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:57.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:56.869+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:57.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:56.870+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:57.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:57.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:57.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:57.273+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:58.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:57.877+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:57.877+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:57.877+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:57.877+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:58.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:57.828+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:58.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:57.828+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:58.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:57.828+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:58.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:57.829+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:58.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:58.320+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:58.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:58.320+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:58.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:58.320+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:58.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:58.321+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:59.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:58.834+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:59.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:58.834+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:59.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:58.834+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:32:59.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:58.835+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:32:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:58.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:58.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:58.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:32:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:58.907+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:32:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:59.283+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:59.283+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:59.283+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:32:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:32:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:32:59.283+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:00.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:59.832+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:00.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:59.832+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:00.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:59.832+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:00.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:32:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:32:59.832+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:59.919+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:59.919+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:59.919+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:32:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:32:59.919+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:00.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:00.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:00.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:00.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:00.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:00.292+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:00.946 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:33:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:33:00.608+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15042 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:33:00.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:00.848+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:00.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:00.848+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:00.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:00.848+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:00.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:00.849+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:01.303 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:00.945+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:01.303 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:00.945+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:01.303 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:00.945+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:01.303 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:00.946+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:01.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:01.302+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:01.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:01.302+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:01.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:01.302+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:01.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:01.303+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:01.909 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:01.806+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:01.910 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:01.806+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:01.910 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:01.806+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:01.910 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:01.807+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:02.302 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:01.909+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:02.302 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:01.909+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:02.302 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:01.909+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:02.302 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:01.909+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:02.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:02.301+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:02.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:02.301+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:02.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:02.301+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:02.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:02.302+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:03.271 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:02.848+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:03.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:02.848+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:03.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:02.848+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:03.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:02.848+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:03.272 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:02.861+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:03.272 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:02.861+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:03.272 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:02.861+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:03.272 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:02.861+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:03.575 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:03.271+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:03.575 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:03.271+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:03.575 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:03.271+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:03.575 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:03.272+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:03.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:03.804+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:03.804+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:03.804+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:03.804+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:04.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:03.903+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:04.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:03.903+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:04.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:03.903+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:04.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:03.904+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:04.293+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:04.293+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:04.293+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:04.293+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:04.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:04.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:04.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:04.828+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:04.870+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:04.870+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:04.870+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:04.871+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:05.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:05.246+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:05.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:05.246+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:05.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:05.246+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:05.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:05.247+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:05.917 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:33:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:33:05.611+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15132 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:33:05.917 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:05.819+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:05.917 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:05.819+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:05.917 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:05.819+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:05.917 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:05.820+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:06.272 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:05.917+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:06.272 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:05.917+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:06.272 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:05.917+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:06.272 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:05.918+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:06.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:06.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:06.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:06.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:06.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:06.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:06.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:06.272+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:07.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:06.789+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:06.789+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:06.789+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:06.790+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:06.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:06.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:06.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:06.958+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:07.311+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:07.311+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:07.311+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:07.312+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:08.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:07.921+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:08.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:07.921+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:07.921+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:07.922+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:07.773+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:07.773+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:07.773+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:07.774+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:08.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:08.292+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:08.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:08.292+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:08.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:08.292+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:08.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:08.293+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:09.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:08.736+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:08.736+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:08.736+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:08.736+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:08.898+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:08.898+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:08.898+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:08.899+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:09.264+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:09.264+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:09.264+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:09.264+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:10.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:09.692+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:10.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:09.692+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:10.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:09.692+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:10.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:09.693+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:09.870+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:09.870+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:09.870+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:10.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:09.870+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:10.224+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:10.224+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:10.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:10.224+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:10.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:10.225+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:10.919 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:33:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:33:10.612+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15219 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:33:10.919 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:10.727+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:10.919 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:10.727+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:10.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:10.727+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:10.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:10.727+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:11.186 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:10.918+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:11.186 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:10.918+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:11.186 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:10.918+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:11.186 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:10.919+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:11.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:11.185+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:11.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:11.185+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:11.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:11.185+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:11.185+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:11.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:11.746+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:11.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:11.746+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:11.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:11.746+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:11.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:11.747+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:12.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:12.187+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:12.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:12.187+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:12.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:12.187+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:12.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:12.188+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:12.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:11.945+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:12.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:11.945+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:12.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:11.945+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:12.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:11.946+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:13.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:12.708+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:13.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:12.708+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:13.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:12.708+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:12.709+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:13.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:12.936+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:13.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:12.936+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:13.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:12.936+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:13.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:12.937+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:13.575 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:13.170+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:13.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:13.170+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:13.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:13.170+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:13.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:13.170+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:13.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:13.704+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:13.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:13.704+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:13.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:13.704+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:13.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:13.705+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:14.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:13.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:14.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:13.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:14.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:13.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:14.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:13.958+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:14.121+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:14.121+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:14.121+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:14.122+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:14.983 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:14.691+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:14.983 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:14.691+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:14.984 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:14.691+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:14.984 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:14.693+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:15.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:15.105+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:15.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:15.105+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:15.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:15.105+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:15.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:15.106+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:15.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:14.982+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:15.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:14.982+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:15.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:14.982+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:15.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:14.983+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:16.012 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:33:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:33:15.613+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15310 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:33:16.012 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:15.737+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:16.012 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:15.737+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:16.012 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:15.737+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:16.012 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:15.738+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:16.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:16.152+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:16.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:16.152+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:16.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:16.152+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:16.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:16.153+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:16.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:16.010+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:16.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:16.010+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:16.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:16.010+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:16.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:16.011+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:17.037 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:16.705+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:17.037 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:16.705+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:17.037 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:16.705+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:17.037 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:16.706+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:17.035+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:17.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:17.035+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:17.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:17.035+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:17.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:17.036+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:17.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:17.145+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:17.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:17.145+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:17.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:17.145+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:17.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:17.146+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:17.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:17.661+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:17.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:17.661+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:17.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:17.661+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:17.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:17.661+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:18.119 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:18.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:18.119 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:18.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:18.119 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:18.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:18.119 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:18.001+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:18.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:18.118+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:18.118+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:18.118+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:18.119+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:19.013 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:18.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:19.013 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:18.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:19.013 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:18.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:19.013 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:18.628+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:19.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:19.149+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:19.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:19.149+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:19.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:19.149+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:19.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:19.150+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:19.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:19.012+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:19.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:19.012+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:19.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:19.012+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:19.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:19.013+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:20.037 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:19.600+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:20.037 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:19.600+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:20.037 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:19.600+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:20.038 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:19.601+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:20.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:20.044+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:20.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:20.044+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:20.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:20.044+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:20.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:20.045+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:20.176+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:20.176+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:20.176+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:20.177+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:20.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:33:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:33:20.615+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15398 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:33:20.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:20.634+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:20.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:20.634+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:20.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:20.634+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:20.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:20.634+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:21.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:21.200+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:21.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:21.200+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:21.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:21.200+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:21.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:21.200+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:21.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:21.054+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:21.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:21.054+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:21.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:21.054+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:21.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:21.055+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:22.071 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:21.672+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:22.071 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:21.672+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:22.071 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:21.672+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:22.071 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:21.673+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:22.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:22.224+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:22.224+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:22.224+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:22.224+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:22.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:22.070+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:22.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:22.070+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:22.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:22.070+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:22.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:22.070+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:23.052 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:22.639+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:23.052 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:22.639+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:23.052 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:22.639+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:23.052 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:22.639+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:23.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:23.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:23.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:23.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:23.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:23.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:23.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:23.052+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:23.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:23.236+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:23.341 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:23.236+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:23.341 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:23.236+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:23.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:23.237+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:23.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:23.597+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:23.597+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:23.597+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:23.598+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:24.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:24.030+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:24.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:24.030+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:24.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:24.030+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:24.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:24.031+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:24.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:24.202+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:24.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:24.202+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:24.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:24.202+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:24.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:24.202+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:24.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:24.579+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:24.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:24.579+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:24.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:24.579+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:24.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:24.580+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:25.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:25.190+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:25.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:25.190+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:25.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:25.190+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:25.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:25.191+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:25.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:25.078+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:25.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:25.078+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:25.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:25.078+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:25.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:25.079+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:26.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:33:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:33:25.617+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15489 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:33:26.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:25.610+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:26.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:25.610+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:26.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:25.610+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:26.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:25.612+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:26.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:26.234+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:26.234+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:26.234+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:26.235+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:26.120+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:26.120+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:26.120+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:26.120+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:27.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:26.636+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:27.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:26.636+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:27.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:26.636+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:27.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:26.637+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:27.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:27.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:27.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:27.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:27.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:27.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:27.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:27.162+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:27.274+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:27.274+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:27.274+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:27.275+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:27.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:27.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:27.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:28.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:27.607+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:28.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:28.281+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:28.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:28.281+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:28.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:28.281+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:28.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:28.282+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:28.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:28.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:28.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:28.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:28.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:28.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:28.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:28.163+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:29.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:28.622+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:28.622+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:28.622+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:28.623+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:29.278+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:29.278+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:29.278+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:29.278+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:29.128+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:29.128+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:29.128+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:29.128+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:30.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:29.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:30.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:29.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:30.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:29.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:30.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:29.627+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:30.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:30.324+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:30.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:30.324+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:30.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:30.324+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:30.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:30.325+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:30.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:30.092+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:30.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:30.092+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:30.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:30.092+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:30.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:30.093+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:31.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:33:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:33:30.618+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15577 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:33:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:30.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:30.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:30.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:30.604+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:31.354 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:31.112+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:31.354 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:31.112+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:31.354 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:31.112+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:31.354 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:31.113+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:31.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:31.353+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:31.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:31.353+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:31.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:31.353+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:31.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:31.354+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:32.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:31.631+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:31.631+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:31.631+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:31.632+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:32.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:32.342+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:32.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:32.342+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:32.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:32.342+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:32.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:32.342+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:32.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:32.103+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:32.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:32.103+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:32.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:32.103+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:32.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:32.104+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:33.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:32.662+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:33.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:32.662+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:33.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:32.662+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:33.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:32.663+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:33.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:33.074+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:33.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:33.074+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:33.074+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:33.075+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:33.689 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:33.365+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:33.689 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:33.365+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:33.689 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:33.365+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:33.689 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:33.366+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:34.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:33.688+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:34.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:33.688+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:34.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:33.688+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:34.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:33.689+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:34.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:34.061+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:34.061+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:34.061+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:34.062+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:34.677 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:34.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:34.677 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:34.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:34.677 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:34.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:34.678 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:34.405+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:35.039 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:34.676+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:35.039 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:34.676+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:35.039 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:34.676+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:35.039 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:34.677+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:35.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:35.038+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:35.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:35.038+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:35.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:35.038+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:35.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:35.039+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:35.621 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:35.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:35.622 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:35.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:35.622 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:35.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:35.622 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:35.360+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:35.908 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:33:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:33:35.620+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15669 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:33:35.909 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:35.702+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:35.909 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:35.702+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:35.909 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:35.702+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:35.909 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:35.704+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:36.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:35.996+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:36.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:35.996+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:36.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:35.996+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:36.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:35.996+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:36.732 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:36.345+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:36.732 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:36.345+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:36.732 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:36.345+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:36.733 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:36.346+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:37.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:36.731+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:37.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:36.731+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:37.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:36.731+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:37.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:36.732+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:36.963+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:36.963+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:36.963+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:36.963+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:37.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:37.342+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:37.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:37.342+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:37.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:37.342+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:37.726 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:37.342+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:38.006 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:37.724+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:38.006 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:37.724+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:38.006 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:37.724+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:38.007 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:37.724+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:38.310 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:38.005+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:38.310 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:38.005+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:38.310 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:38.005+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:38.310 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:38.006+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:38.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:38.309+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:38.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:38.309+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:38.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:38.309+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:38.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:38.310+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:38.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:38.709+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:38.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:38.709+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:38.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:38.709+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:38.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:38.710+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:39.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:38.972+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:39.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:38.972+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:39.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:38.972+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:39.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:38.972+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:39.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:39.308+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:39.308+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:39.308+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:39.308+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:40.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:39.734+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:39.734+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:39.734+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:39.735+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:39.951+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:39.951+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:39.951+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:39.952+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:40.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:40.261+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:40.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:40.261+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:40.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:40.261+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:40.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:40.262+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:40.999 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:33:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:33:40.622+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15757 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:33:41.000 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:40.778+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:41.000 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:40.778+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:41.000 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:40.778+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:41.000 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:40.779+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:41.268 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:40.998+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:41.268 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:40.998+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:41.268 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:40.998+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:41.268 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:40.999+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:41.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:41.267+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:41.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:41.267+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:41.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:41.267+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:41.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:41.268+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:41.998 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:41.735+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:41.998 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:41.735+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:41.998 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:41.735+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:41.999 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:41.735+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:42.250 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:41.997+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:42.250 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:41.997+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:42.251 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:41.997+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:42.251 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:41.998+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:42.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:42.249+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:42.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:42.249+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:42.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:42.249+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:42.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:42.250+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:43.032 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:42.760+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:43.032 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:42.760+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:43.032 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:42.760+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:43.032 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:42.761+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:43.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:43.205+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:43.205+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:43.205+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:43.206+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:43.031+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:43.031+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:43.031+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:43.032+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:43.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:43.787+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:43.787+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:43.787+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:43.788+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:43.993+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:43.993+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:43.993+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:43.994+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:44.232+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:44.232+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:44.232+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:44.233+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:45.019 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:44.743+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:45.019 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:44.743+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:45.019 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:44.743+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:45.019 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:44.744+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:45.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:45.228+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:45.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:45.228+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:45.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:45.228+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:45.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:45.228+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:45.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:45.018+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:45.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:45.018+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:45.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:45.018+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:45.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:45.019+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:46.051 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:33:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:33:45.624+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15848 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:33:46.052 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:45.732+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:46.052 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:45.732+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:46.052 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:45.732+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:46.052 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:45.732+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:46.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:46.191+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:46.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:46.191+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:46.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:46.191+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:46.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:46.192+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:46.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:46.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:46.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:46.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:46.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:46.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:46.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:46.051+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:47.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:46.750+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:47.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:46.750+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:47.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:46.750+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:47.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:46.751+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:47.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:47.202+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:47.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:47.202+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:47.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:47.202+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:47.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:47.203+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:47.090+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:47.090+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:47.090+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:47.090+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:48.050 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:47.777+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:48.050 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:47.777+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:48.050 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:47.777+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:48.050 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:47.778+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:48.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:48.182+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:48.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:48.182+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:48.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:48.182+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:48.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:48.183+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:48.049+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:48.049+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:48.049+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:48.049+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:48.784+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:48.784+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:49.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:48.784+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:49.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:48.785+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:49.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:49.093+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:49.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:49.093+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:49.093+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:49.093+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:49.181+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:49.181+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:49.181+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:49.182+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:49.825+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:49.825+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:49.825+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:49.826+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:50.055+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:50.055+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:50.055+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:50.056+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:50.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:50.206+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:50.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:50.206+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:50.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:50.206+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:50.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:50.207+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:51.023 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:33:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:33:50.625+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15936 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:33:51.023 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:50.808+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:51.023 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:50.808+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:51.023 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:50.808+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:51.023 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:50.809+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:51.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:51.200+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:51.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:51.200+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:51.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:51.200+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:51.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:51.202+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:51.022+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:51.022+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:51.022+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:51.023+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:52.034 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:51.771+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:52.034 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:51.771+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:52.034 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:51.771+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:52.034 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:51.772+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:52.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:52.234+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:52.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:52.234+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:52.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:52.234+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:52.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:52.235+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:52.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:52.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:52.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:52.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:52.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:52.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:52.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:52.034+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:53.004+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:53.004+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:53.004+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:53.004+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:53.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:52.775+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:53.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:52.775+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:53.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:52.775+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:53.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:52.776+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:53.579 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:53.278+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:53.579 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:53.278+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:53.579 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:53.278+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:53.579 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:53.279+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:53.765+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:53.765+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:53.765+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:53.766+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:54.316 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:53.996+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:54.316 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:53.997+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:54.316 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:53.997+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:54.316 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:53.997+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:54.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:54.315+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:54.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:54.315+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:54.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:54.315+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:54.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:54.316+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:55.023 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:54.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:55.023 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:54.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:55.023 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:54.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:55.023 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:54.763+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:55.286 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:55.022+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:55.286 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:55.022+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:55.286 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:55.022+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:55.286 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:55.023+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:55.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:55.285+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:55.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:55.285+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:55.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:55.285+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:55.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:55.286+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:55.996 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:33:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:33:55.627+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16028 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:33:55.996 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:55.752+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:55.996 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:55.752+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:55.996 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:55.752+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:55.996 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:55.753+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:56.294 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:55.995+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:56.294 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:55.995+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:56.294 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:55.995+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:56.294 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:55.996+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:56.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:56.292+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:56.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:56.292+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:56.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:56.293+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:56.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:56.294+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:57.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:56.753+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:57.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:56.753+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:57.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:56.754+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:57.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:56.754+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:57.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:56.960+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:57.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:56.960+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:57.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:56.960+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:57.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:56.961+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:57.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:57.287+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:57.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:57.287+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:57.287+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:57.287+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:58.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:57.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:58.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:57.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:58.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:57.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:58.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:57.792+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:57.993+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:57.993+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:57.993+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:57.994+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:58.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:58.282+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:58.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:58.282+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:58.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:58.282+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:58.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:58.283+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:59.040 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:58.784+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:59.040 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:58.784+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:59.041 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:58.784+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:33:59.041 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:58.785+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:33:59.333 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:59.039+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:59.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:59.039+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:59.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:59.039+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:33:59.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:33:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:33:59.040+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:33:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:59.332+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:59.332+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:59.332+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:33:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:33:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:33:59.334+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:00.018 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:59.761+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:00.019 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:59.761+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:00.019 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:59.761+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:00.019 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:33:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:33:59.762+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:00.303 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:00.017+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:00.304 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:00.017+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:00.304 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:00.017+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:00.304 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:00.018+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:00.302+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:00.302+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:00.302+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:00.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:00.303+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:01.034 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:34:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:34:00.629+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16116 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:34:01.034 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:00.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:01.034 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:00.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:01.034 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:00.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:01.034 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:00.763+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:01.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:01.256+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:01.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:01.256+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:01.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:01.256+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:01.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:01.257+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:01.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:01.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:01.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:01.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:01.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:01.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:01.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:01.034+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:01.998 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:01.718+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:01.998 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:01.718+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:01.998 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:01.718+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:01.998 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:01.719+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:02.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:01.996+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:01.996+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:01.996+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:01.997+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:02.235+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:02.235+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:02.235+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:02.236+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:03.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:02.749+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:02.749+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:02.749+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:02.750+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:02.975+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:02.975+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:02.975+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:02.976+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:03.556 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:03.276+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:03.556 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:03.276+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:03.556 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:03.276+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:03.556 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:03.276+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:03.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:03.755+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:03.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:03.755+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:03.755+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:03.756+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:04.273 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:03.942+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:04.273 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:03.942+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:04.274 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:03.942+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:04.274 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:03.942+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:04.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:04.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:04.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:04.273+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:04.749+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:04.749+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:04.749+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:04.750+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:04.892+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:04.892+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:04.893+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:04.894+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:05.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:05.262+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:05.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:05.262+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:05.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:05.262+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:05.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:05.263+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:05.932 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:34:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:34:05.631+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16207 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:34:05.933 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:05.729+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:05.933 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:05.729+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:05.933 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:05.729+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:05.933 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:05.729+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:06.250 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:05.931+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:06.250 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:05.931+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:06.250 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:05.931+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:06.250 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:05.932+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:06.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:06.249+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:06.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:06.249+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:06.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:06.249+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:06.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:06.250+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:06.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:06.749+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:06.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:06.749+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:06.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:06.749+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:06.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:06.750+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:07.291 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:06.970+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:07.291 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:06.970+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:07.291 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:06.970+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:07.291 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:06.971+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:07.290+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:07.290+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:07.290+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:07.290+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:08.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:07.729+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:08.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:07.729+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:07.729+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:07.730+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:07.971+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:07.971+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:07.971+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:07.972+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:08.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:08.304+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:08.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:08.304+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:08.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:08.304+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:08.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:08.306+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:08.713+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:08.713+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:08.713+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:08.714+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:08.925+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:08.925+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:08.925+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:08.926+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:09.295+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:09.295+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:09.295+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:09.296+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:10.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:09.877+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:10.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:09.877+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:09.877+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:09.877+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:10.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:09.677+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:10.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:09.677+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:10.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:09.677+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:10.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:09.678+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:10.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:10.274+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:10.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:10.274+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:10.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:10.274+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:10.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:10.274+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:10.914 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:34:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:34:10.632+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16295 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:34:10.915 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:10.711+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:10.915 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:10.711+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:10.915 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:10.711+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:10.915 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:10.711+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:11.262 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:10.913+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:11.262 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:10.913+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:11.262 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:10.913+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:11.262 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:10.914+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:11.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:11.261+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:11.261+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:11.261+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:11.262+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:11.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:12.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:11.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:12.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:11.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:12.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:11.930+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:11.698+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:11.698+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:11.698+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:12.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:11.699+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:12.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:12.276+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:12.276+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:12.276+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:12.277+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:13.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:12.746+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:12.746+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:12.746+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:12.746+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:13.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:12.883+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:13.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:12.883+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:13.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:12.883+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:13.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:12.884+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:13.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:13.278+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:13.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:13.278+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:13.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:13.278+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:13.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:13.279+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:13.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:13.768+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:13.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:13.768+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:13.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:13.768+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:13.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:13.768+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:14.277 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:13.844+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:14.278 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:13.844+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:14.278 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:13.844+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:14.278 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:13.844+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:14.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:14.276+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:14.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:14.276+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:14.276+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:14.277+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:15.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:14.786+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:14.787+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:14.787+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:14.787+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:14.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:14.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:14.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:14.848+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:15.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:15.252+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:15.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:15.252+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:15.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:15.252+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:15.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:15.253+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:16.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:34:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:34:15.634+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16387 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:34:16.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:15.880+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:16.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:15.880+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:16.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:15.880+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:16.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:15.880+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:15.739+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:15.739+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:15.739+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:15.740+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:16.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:16.240+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:16.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:16.240+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:16.240+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:16.241+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:17.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:16.767+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:16.767+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:16.767+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:16.768+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:17.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:16.835+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:17.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:16.835+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:17.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:16.835+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:17.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:16.835+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:17.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:17.262+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:17.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:17.262+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:17.262+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:17.263+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:18.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:17.812+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:17.812+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:17.812+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:17.813+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:18.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:17.874+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:18.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:17.874+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:18.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:17.874+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:18.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:17.875+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:18.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:18.310+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:18.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:18.310+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:18.310+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:18.311+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:19.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:18.842+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:19.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:18.842+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:19.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:18.842+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:19.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:18.842+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:19.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:18.853+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:19.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:18.853+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:19.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:18.853+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:19.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:18.853+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:19.301+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:19.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:19.301+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:19.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:19.301+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:19.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:19.303+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:20.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:19.822+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:20.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:19.822+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:20.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:19.822+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:20.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:19.822+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:20.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:19.819+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:20.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:19.819+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:20.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:19.819+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:20.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:19.820+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:20.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:20.287+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:20.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:20.287+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:20.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:20.287+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:20.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:20.288+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:21.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:34:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:34:20.636+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16475 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:34:21.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:20.773+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:21.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:20.773+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:21.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:20.773+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:21.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:20.774+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:21.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:20.797+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:21.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:20.797+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:21.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:20.797+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:21.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:20.797+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:21.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:21.306+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:21.306+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:21.306+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:21.306+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:21.988 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:21.772+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:21.988 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:21.772+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:21.988 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:21.772+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:21.988 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:21.772+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:21.989 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:21.766+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:21.989 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:21.766+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:21.989 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:21.766+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:21.989 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:21.768+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:22.756 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:22.343+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:22.756 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:22.343+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:22.756 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:22.343+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:22.756 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:22.345+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:22.804+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:22.804+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:22.804+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:23.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:22.805+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:22.756+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:22.756+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:22.756+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:22.757+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:23.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:23.343+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:23.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:23.343+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:23.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:23.343+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:23.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:23.344+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:24.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:23.787+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:24.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:23.787+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:24.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:23.787+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:24.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:23.788+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:23.785+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:23.785+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:23.785+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:23.786+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:24.745 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:24.373+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:24.746 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:24.373+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:24.746 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:24.373+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:24.746 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:24.374+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:25.060 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:24.745+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:25.060 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:24.745+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:25.060 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:24.745+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:25.060 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:24.746+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:25.060 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:24.772+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:25.060 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:24.772+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:25.060 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:24.772+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:25.061 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:24.773+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:25.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:25.327+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:25.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:25.327+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:25.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:25.327+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:25.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:25.328+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:26.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:34:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:34:25.639+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16566 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:34:26.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:25.787+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:26.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:25.787+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:26.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:25.787+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:26.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:25.788+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:25.709+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:25.709+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:25.709+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:25.709+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:26.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:26.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:26.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:26.326+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:27.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:26.779+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:27.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:26.779+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:27.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:26.779+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:27.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:26.780+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:27.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:26.735+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:27.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:26.735+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:27.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:26.735+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:27.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:26.735+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:27.329+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:27.329+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:27.329+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:27.330+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:27.772+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:28.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:27.772+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:28.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:27.772+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:28.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:27.773+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:27.784+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:27.784+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:27.784+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:27.784+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:28.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:28.377+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:28.726 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:28.377+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:28.726 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:28.377+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:28.726 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:28.378+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:29.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:28.725+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:28.725+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:28.725+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:28.726+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:29.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:28.738+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:29.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:28.738+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:29.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:28.738+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:29.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:28.738+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:29.386+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:29.386+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:29.386+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:29.387+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:30.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:29.749+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:30.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:29.749+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:30.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:29.749+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:30.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:29.750+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:30.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:29.784+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:30.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:29.784+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:30.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:29.784+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:30.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:29.784+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:30.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:30.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:30.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:30.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:30.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:30.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:30.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:30.397+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:31.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:34:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:34:30.639+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16654 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:34:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:30.778+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:30.778+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:30.778+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:30.779+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:30.753+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:30.753+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:30.753+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:30.754+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:31.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:31.426+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:31.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:31.426+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:31.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:31.426+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:31.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:31.426+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:32.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:31.814+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:32.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:31.814+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:31.815+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:31.815+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:31.795+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:31.795+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:31.795+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:31.796+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:32.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:32.437+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:32.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:32.437+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:32.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:32.437+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:32.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:32.438+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:33.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:32.842+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:33.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:32.842+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:33.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:32.842+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:33.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:32.843+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:32.842+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:32.842+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:32.842+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:32.843+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:33.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:33.446+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:33.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:33.446+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:33.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:33.446+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:33.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:33.447+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:34.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:33.823+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:34.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:33.823+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:34.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:33.823+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:34.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:33.824+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:34.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:33.841+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:34.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:33.841+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:34.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:33.841+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:34.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:33.842+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:34.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:34.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:34.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:34.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:34.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:34.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:34.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:34.401+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:35.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:34.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:35.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:34.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:35.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:34.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:35.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:34.792+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:35.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:34.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:35.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:34.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:35.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:34.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:35.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:34.819+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:35.642 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:35.386+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:35.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:35.386+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:35.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:35.386+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:35.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:35.387+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:36.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:34:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:34:35.641+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16746 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:34:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:35.798+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:35.799+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:35.799+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:35.800+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:35.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:35.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:35.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:35.861+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:36.794 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:36.421+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:36.794 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:36.421+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:36.794 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:36.421+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:36.794 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:36.422+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:36.814+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:36.814+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:36.814+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:36.815+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:37.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:36.793+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:37.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:36.793+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:37.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:36.793+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:37.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:36.793+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:37.759 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:37.387+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:37.759 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:37.387+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:37.759 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:37.387+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:37.759 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:37.388+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:38.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:37.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:38.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:37.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:38.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:37.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:38.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:37.776+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:38.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:37.758+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:38.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:37.758+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:38.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:37.758+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:38.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:37.759+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:38.737 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:38.381+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:38.738 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:38.381+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:38.738 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:38.381+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:38.738 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:38.382+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:39.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:38.815+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:38.815+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:38.815+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:38.816+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:38.736+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:38.736+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:38.736+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:38.738+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:39.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:39.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:39.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:39.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:39.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:39.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:39.396+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:39.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:39.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:39.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:39.763+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:39.843+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:39.843+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:39.843+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:39.844+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:40.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:40.377+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:40.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:40.377+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:40.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:40.377+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:40.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:40.378+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:40.764 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:34:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:34:40.643+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16834 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:34:41.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:40.763+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:41.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:40.763+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:41.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:40.763+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:41.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:40.764+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:41.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:40.887+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:41.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:40.887+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:41.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:40.887+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:41.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:40.888+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:41.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:41.378+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:41.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:41.378+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:41.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:41.378+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:41.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:41.379+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:42.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:41.754+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:42.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:41.754+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:42.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:41.754+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:42.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:41.756+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:42.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:41.885+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:42.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:41.885+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:42.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:41.885+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:42.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:41.886+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:42.769 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:42.422+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:42.769 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:42.422+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:42.769 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:42.422+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:42.769 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:42.424+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:43.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:42.768+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:43.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:42.768+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:43.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:42.768+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:43.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:42.768+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:43.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:42.868+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:43.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:42.868+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:43.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:42.868+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:43.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:42.868+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:43.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:43.391+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:43.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:43.391+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:43.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:43.391+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:43.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:43.392+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:44.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:43.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:44.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:43.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:44.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:43.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:44.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:43.794+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:44.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:43.914+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:44.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:43.914+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:44.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:43.914+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:44.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:43.914+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:44.766 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:44.413+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:44.767 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:44.413+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:44.767 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:44.413+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:44.767 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:44.414+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:44.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:44.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:44.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:44.930+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:45.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:44.765+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:45.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:44.765+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:45.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:44.765+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:45.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:44.766+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:45.646 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:45.391+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:45.646 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:45.391+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:45.646 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:45.391+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:45.646 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:45.392+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:45.903 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:34:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:34:45.645+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16925 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:34:45.903 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:45.735+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:45.903 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:45.735+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:45.903 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:45.735+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:45.903 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:45.735+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:45.904 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:45.902+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:45.904 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:45.902+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:45.904 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:45.902+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:46.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:45.903+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:46.743 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:46.426+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:46.743 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:46.426+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:46.743 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:46.426+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:46.743 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:46.427+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:47.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:46.862+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:46.862+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:46.862+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:46.863+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:47.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:46.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:47.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:46.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:47.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:46.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:47.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:46.743+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:47.748 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:47.410+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:47.748 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:47.410+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:47.749 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:47.410+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:47.749 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:47.410+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:48.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:47.747+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:48.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:47.747+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:48.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:47.747+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:48.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:47.748+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:47.850+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:47.850+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:47.850+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:47.851+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:48.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:48.382+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:48.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:48.382+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:48.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:48.382+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:48.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:48.383+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:48.699+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:49.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:48.699+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:49.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:48.699+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:49.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:48.700+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:49.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:48.898+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:49.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:48.898+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:49.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:48.898+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:49.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:48.898+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:49.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:49.389+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:49.389+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:49.389+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:49.390+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:49.698+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:49.698+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:49.698+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:49.699+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:49.891+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:49.891+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:49.891+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:49.892+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:50.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:50.404+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:50.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:50.404+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:50.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:50.404+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:50.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:50.404+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:50.935 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:34:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:34:50.647+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17013 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:34:50.935 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:50.702+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:50.935 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:50.702+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:50.935 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:50.702+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:50.935 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:50.703+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:51.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:50.934+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:50.934+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:50.934+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:50.935+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:51.696 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:51.439+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:51.696 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:51.439+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:51.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:51.439+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:51.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:51.440+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:51.894+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:51.894+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:51.894+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:51.895+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:51.715+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:51.715+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:51.715+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:51.716+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:52.708 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:52.452+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:52.708 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:52.452+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:52.708 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:52.452+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:52.708 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:52.453+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:52.709 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:52.667+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:52.709 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:52.667+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:52.709 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:52.667+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:52.709 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:52.667+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:52.898+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:52.898+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:52.898+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:52.899+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:53.733 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:53.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:53.733 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:53.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:53.733 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:53.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:53.733 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:53.468+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:53.733 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:53.647+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:53.734 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:53.647+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:53.734 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:53.647+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:53.734 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:53.648+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:54.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:53.926+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:54.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:53.926+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:54.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:53.926+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:54.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:53.927+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:54.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:54.439+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:54.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:54.439+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:54.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:54.439+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:54.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:54.440+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:54.634+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:54.634+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:54.634+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:54.635+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:55.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:54.959+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:55.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:54.959+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:55.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:54.959+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:55.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:54.960+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:55.774 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:34:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:34:55.649+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17105 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:34:55.775 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:55.651+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:55.775 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:55.651+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:55.775 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:55.651+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:55.775 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:55.652+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:55.776 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:55.451+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:55.776 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:55.451+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:55.776 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:55.451+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:55.776 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:55.452+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:56.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:55.947+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:56.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:55.947+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:56.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:55.947+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:56.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:55.948+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:56.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:56.500+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:56.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:56.500+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:56.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:56.500+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:56.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:56.501+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:56.700+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:56.700+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:56.700+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:56.700+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:57.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:56.898+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:56.898+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:56.898+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:56.898+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:57.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:57.707+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:57.707+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:57.707+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:57.707+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:57.492+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:57.492+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:57.492+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:57.493+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:57.917+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:58.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:57.917+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:58.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:57.917+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:58.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:57.918+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:58.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:58.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:58.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:58.455+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:58.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:58.658+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:58.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:58.658+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:58.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:58.658+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:34:58.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:58.658+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:34:59.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:58.883+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:59.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:58.883+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:59.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:58.883+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:34:59.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:58.884+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:34:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:59.413+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:59.413+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:59.413+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:34:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:34:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:34:59.414+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:00.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:59.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:59.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:59.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:34:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:34:59.907+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:00.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:59.688+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:00.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:59.688+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:00.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:59.688+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:00.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:34:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:34:59.689+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:00.378+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:00.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:00.378+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:00.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:00.378+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:00.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:00.379+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:00.935 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:35:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:35:00.650+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17193 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:35:00.936 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:00.726+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:00.936 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:00.726+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:00.936 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:00.726+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:00.936 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:00.727+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:00.934+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:01.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:00.934+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:01.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:00.934+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:01.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:00.935+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:01.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:01.352+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:01.747 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:01.352+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:01.748 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:01.352+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:01.748 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:01.353+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:01.706+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:01.706+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:01.706+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:01.707+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:02.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:01.939+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:02.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:01.939+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:02.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:01.939+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:02.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:01.940+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:02.743 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:02.395+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:02.743 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:02.395+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:02.743 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:02.395+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:02.743 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:02.396+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:03.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:02.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:02.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:02.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:02.743+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:02.893+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:02.893+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:02.893+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:02.894+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:03.763 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:03.363+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:03.763 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:03.363+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:03.763 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:03.363+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:03.763 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:03.364+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:04.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:03.855+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:04.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:03.855+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:04.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:03.855+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:04.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:03.855+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:04.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:03.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:04.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:03.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:04.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:03.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:04.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:03.763+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:04.331+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:04.331+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:04.331+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:04.332+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:05.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:04.846+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:04.846+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:04.846+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:04.847+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:04.812+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:04.812+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:04.812+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:04.813+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:05.653 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:05.374+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:05.654 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:05.374+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:05.654 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:05.374+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:05.654 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:05.375+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:06.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:35:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:35:05.652+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17283 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:35:06.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:05.832+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:06.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:05.832+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:06.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:05.832+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:06.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:05.833+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:05.826+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:05.826+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:05.826+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:05.826+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:06.790 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:06.383+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:06.790 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:06.383+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:06.790 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:06.383+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:06.790 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:06.384+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:06.789+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:06.789+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:06.789+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:06.789+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:06.841+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:06.841+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:06.841+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:06.841+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:07.809 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:07.414+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:07.809 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:07.414+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:07.810 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:07.414+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:07.810 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:07.415+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:07.808+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:07.808+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:07.808+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:07.809+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:08.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:07.854+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:08.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:07.854+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:08.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:07.854+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:08.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:07.855+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:08.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:08.424+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:08.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:08.424+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:08.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:08.424+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:08.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:08.425+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:09.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:08.850+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:08.850+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:08.850+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:08.851+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:08.823+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:08.823+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:08.823+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:08.824+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:09.787 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:09.383+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:09.788 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:09.383+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:09.788 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:09.383+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:09.788 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:09.384+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:09.786+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:09.786+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:09.786+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:09.787+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:10.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:09.888+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:10.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:09.888+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:10.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:09.888+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:10.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:09.889+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:10.680 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:35:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:35:10.654+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17372 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:35:10.680 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:10.422+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:10.680 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:10.422+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:10.680 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:10.422+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:10.680 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:10.423+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:10.865+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:10.865+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:11.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:10.865+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:11.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:10.866+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:10.768+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:10.769+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:10.769+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:10.769+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:11.762 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:11.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:11.762 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:11.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:11.763 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:11.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:11.763 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:11.450+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:11.761+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:11.761+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:12.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:11.761+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:12.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:11.762+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:11.867+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:11.867+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:11.867+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:11.868+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:12.792 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:12.483+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:12.793 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:12.483+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:12.793 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:12.483+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:12.793 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:12.484+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:12.791+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:12.791+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:13.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:12.791+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:13.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:12.792+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:12.821+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:12.821+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:12.821+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:12.822+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:13.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:13.473+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:13.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:13.473+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:13.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:13.473+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:13.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:13.474+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:14.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:13.784+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:13.784+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:13.784+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:13.785+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:14.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:13.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:14.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:13.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:14.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:13.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:14.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:13.828+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:14.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:14.438+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:14.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:14.438+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:14.438+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:14.439+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:15.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:14.848+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:14.848+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:14.848+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:14.849+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:14.799+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:14.799+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:14.799+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:14.800+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:15.779 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:35:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:35:15.656+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17463 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:35:15.780 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:15.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:15.780 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:15.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:15.780 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:15.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:15.780 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:15.480+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:16.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:15.817+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:16.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:15.817+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:15.817+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:15.817+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:16.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:15.838+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:16.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:15.838+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:16.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:15.838+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:16.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:15.839+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:16.802 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:16.511+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:16.802 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:16.511+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:16.803 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:16.511+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:16.803 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:16.511+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:17.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:16.865+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:17.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:16.865+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:16.865+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:16.865+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:17.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:16.800+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:17.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:16.800+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:17.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:16.800+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:17.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:16.801+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:17.535 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:17.533+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:17.535 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:17.533+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:17.535 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:17.533+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:17.536 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:17.534+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:18.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:17.851+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:17.851+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:17.851+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:17.851+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:18.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:17.822+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:18.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:17.822+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:18.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:17.822+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:18.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:17.823+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:18.825 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:18.790+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:18.825 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:18.791+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:18.825 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:18.791+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:18.825 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:18.791+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:18.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:18.544+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:18.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:18.544+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:18.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:18.544+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:18.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:18.545+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:19.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:18.823+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:19.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:18.823+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:19.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:18.823+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:18.825+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:19.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:19.586+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:19.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:19.586+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:19.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:19.586+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:19.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:19.587+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:19.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:19.758+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:19.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:19.758+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:19.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:19.758+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:19.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:19.759+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:20.165 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:19.856+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:20.165 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:19.856+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:20.165 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:19.856+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:20.165 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:19.857+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:20.892 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:35:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:35:20.658+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17552 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:35:20.893 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:20.605+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:20.893 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:20.605+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:20.893 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:20.605+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:20.893 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:20.606+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:20.893 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:20.747+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:20.893 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:20.747+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:20.894 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:20.747+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:20.894 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:20.748+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:21.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:20.892+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:21.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:20.892+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:21.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:20.892+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:21.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:20.893+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:21.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:35:21.587+0000 7f31d1cbb700 0 monclient(hunting): authenticate timed out after 300 2024-03-19T03:35:21.589 INFO:teuthology.orchestra.run.smithi059.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-19T03:35:21.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:21.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:21.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:21.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:21.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:21.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:21.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:21.764+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:21.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:21.563+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:21.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:21.563+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:21.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:21.563+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:21.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:21.563+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:22.325 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:21.870+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:22.325 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:21.870+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:22.325 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:21.870+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:22.325 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:21.871+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:22.886 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:22.597+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:22.886 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:22.597+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:22.886 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:22.597+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:22.887 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:22.597+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:22.887 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:22.738+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:22.887 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:22.738+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:22.887 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:22.738+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:22.887 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:22.739+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:23.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:22.885+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:23.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:22.885+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:23.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:22.885+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:22.886+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:23.715 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:23.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:23.716 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:23.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:23.716 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:23.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:23.716 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:23.572+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:24.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:23.936+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:24.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:23.936+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:24.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:23.936+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:24.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:23.937+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:23.714+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:23.714+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:23.714+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:23.715+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:24.538+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:24.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:24.538+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:24.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:24.538+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:24.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:24.540+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:24.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:24.741+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:24.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:24.741+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:24.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:24.741+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:24.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:24.742+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:25.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:24.975+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:25.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:24.975+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:25.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:24.975+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:25.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:24.976+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:25.786 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:35:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:35:25.660+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17642 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:35:25.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:25.528+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:25.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:25.528+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:25.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:25.528+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:25.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:25.528+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:25.787 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:25.773+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:25.787 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:25.773+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:25.787 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:25.773+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:25.787 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:25.773+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:26.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:25.979+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:26.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:25.979+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:26.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:25.979+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:26.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:25.980+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:26.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:26.512+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:26.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:26.512+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:26.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:26.512+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:26.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:26.513+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:26.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:26.732+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:26.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:26.732+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:26.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:26.732+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:26.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:26.733+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:27.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:26.992+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:27.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:26.992+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:27.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:26.992+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:27.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:26.993+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:27.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:27.749+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:27.749+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:27.749+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:27.749+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:27.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:27.501+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:27.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:27.501+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:27.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:27.501+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:27.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:27.501+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:28.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:27.963+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:28.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:27.963+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:28.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:27.963+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:28.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:27.964+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:28.742 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:28.475+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:28.743 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:28.475+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:28.743 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:28.475+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:28.743 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:28.476+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:29.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:28.986+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:29.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:28.986+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:28.986+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:28.987+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:29.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:28.742+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:29.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:28.742+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:29.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:28.742+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:29.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:28.742+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:29.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:29.501+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:29.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:29.501+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:29.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:29.501+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:29.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:29.502+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:29.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:29.725+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:29.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:29.725+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:29.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:29.725+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:29.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:29.726+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:30.166 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:30.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:30.166 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:30.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:30.166 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:30.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:30.166 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:30.022+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:30.764 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:35:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:35:30.661+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17730 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:35:30.764 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:30.484+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:30.764 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:30.484+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:30.764 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:30.484+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:30.764 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:30.485+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:31.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:30.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:30.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:30.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:30.764+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:30.994+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:30.994+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:30.994+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:30.996+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:31.791 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:31.521+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:31.791 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:31.521+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:31.791 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:31.521+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:31.791 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:31.522+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:32.082 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:31.980+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:32.082 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:31.980+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:32.082 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:31.980+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:32.083 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:31.981+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:32.083 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:31.790+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:32.083 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:31.790+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:32.083 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:31.790+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:32.083 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:31.791+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:32.829 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:32.505+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:32.829 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:32.505+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:32.829 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:32.505+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:32.829 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:32.506+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:32.828+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:33.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:32.828+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:33.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:32.828+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:33.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:32.828+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:33.027+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:33.027+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:33.027+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:33.028+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:33.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:33.527+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:33.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:33.527+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:33.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:33.527+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:33.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:33.528+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:34.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:34.070+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:34.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:34.070+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:34.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:34.070+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:34.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:34.071+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:33.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:33.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:33.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:33.860+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:34.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:34.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:34.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:34.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:34.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:34.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:34.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:34.506+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:35.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:35.079+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:35.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:35.079+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:35.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:35.079+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:35.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:35.079+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:35.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:34.837+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:35.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:34.837+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:35.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:34.837+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:35.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:34.838+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:35.788 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:35:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:35:35.663+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17820 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:35:35.788 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:35.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:35.788 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:35.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:35.788 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:35.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:35.788 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:35.479+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:36.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:35.871+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:36.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:35.871+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:35.871+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:35.872+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:36.481 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:36.129+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:36.481 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:36.129+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:36.481 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:36.129+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:36.482 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:36.130+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:36.836 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:36.480+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:36.836 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:36.480+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:36.836 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:36.480+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:36.836 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:36.481+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:36.835+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:36.835+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:36.835+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:36.836+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:37.495 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:37.167+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:37.495 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:37.167+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:37.495 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:37.167+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:37.495 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:37.168+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:37.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:37.494+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:37.494+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:37.494+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:37.495+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:38.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:37.836+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:38.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:37.836+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:38.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:37.836+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:38.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:37.837+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:38.507 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:38.169+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:38.507 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:38.169+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:38.507 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:38.169+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:38.507 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:38.170+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:38.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:38.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:38.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:38.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:38.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:38.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:38.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:38.507+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:39.141 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:38.844+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:39.142 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:38.844+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:39.142 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:38.844+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:39.142 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:38.844+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:39.510 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:39.141+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:39.510 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:39.141+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:39.510 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:39.141+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:39.510 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:39.141+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:39.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:39.508+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:39.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:39.508+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:39.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:39.508+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:39.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:39.509+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:40.166 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:40.122+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:40.166 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:40.122+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:40.166 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:40.122+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:40.166 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:40.123+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:40.166 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:39.879+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:40.167 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:39.879+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:40.167 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:39.879+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:40.167 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:39.879+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:40.789 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:35:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:35:40.665+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17909 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:35:40.790 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:40.535+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:40.790 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:40.535+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:40.790 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:40.535+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:40.790 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:40.535+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:41.084 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:40.831+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:41.085 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:40.831+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:41.085 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:40.831+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:41.085 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:40.831+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:41.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:41.083+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:41.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:41.083+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:41.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:41.083+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:41.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:41.084+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:41.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:41.565+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:41.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:41.565+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:41.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:41.565+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:41.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:41.567+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:42.142 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:42.087+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:42.142 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:42.087+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:42.142 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:42.089+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:42.142 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:42.090+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:42.143 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:41.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:42.143 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:41.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:42.143 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:41.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:42.143 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:41.858+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:42.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:42.557+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:42.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:42.557+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:42.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:42.557+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:42.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:42.558+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:43.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:43.065+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:43.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:43.065+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:43.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:43.065+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:43.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:43.066+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:43.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:42.888+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:43.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:42.888+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:43.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:42.888+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:43.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:42.889+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:43.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:43.574+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:43.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:43.574+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:43.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:43.574+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:43.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:43.575+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:44.095+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:44.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:44.095+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:44.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:44.095+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:44.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:44.096+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:43.939+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:43.939+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:43.939+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:43.939+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:44.528+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:44.528+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:44.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:44.528+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:44.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:44.529+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:45.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:45.056+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:45.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:45.056+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:45.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:45.056+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:45.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:45.057+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:45.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:44.894+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:45.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:44.894+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:45.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:44.894+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:45.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:44.895+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:45.794 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:35:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:35:45.668+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17999 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:35:45.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:45.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:45.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:45.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:45.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:45.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:45.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:45.511+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:46.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:45.914+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:46.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:45.914+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:46.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:45.914+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:46.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:45.915+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:46.560 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:46.092+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:46.560 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:46.092+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:46.560 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:46.092+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:46.560 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:46.093+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:46.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:46.559+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:46.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:46.559+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:46.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:46.559+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:46.560+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:47.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:47.080+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:47.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:47.080+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:47.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:47.080+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:47.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:47.081+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:47.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:46.953+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:47.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:46.953+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:47.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:46.953+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:47.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:46.954+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:47.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:47.538+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:47.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:47.538+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:47.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:47.538+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:47.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:47.539+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:48.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:48.067+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:48.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:48.067+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:48.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:48.067+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:48.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:48.068+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:48.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:48.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:48.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:48.001+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:48.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:48.527+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:48.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:48.527+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:48.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:48.527+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:48.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:48.528+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:49.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:49.022+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:49.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:49.022+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:49.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:49.022+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:49.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:49.023+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:49.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:49.041+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:49.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:49.041+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:49.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:49.041+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:49.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:49.041+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:49.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:49.531+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:49.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:49.531+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:49.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:49.531+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:49.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:49.532+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:50.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:50.020+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:50.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:50.020+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:50.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:50.020+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:50.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:50.020+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:50.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:50.024+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:50.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:50.024+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:50.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:50.024+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:50.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:50.024+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:50.794 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:35:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:35:50.669+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18088 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:35:50.794 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:50.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:50.794 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:50.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:50.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:50.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:50.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:50.489+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:51.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:51.074+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:51.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:51.074+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:51.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:51.074+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:51.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:51.075+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:51.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:51.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:51.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:51.058+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:51.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:51.493+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:51.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:51.493+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:51.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:51.493+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:51.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:51.494+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:52.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:52.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:52.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:52.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:52.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:52.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:52.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:52.032+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:52.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:52.096+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:52.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:52.096+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:52.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:52.096+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:52.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:52.097+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:52.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:52.544+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:52.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:52.544+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:52.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:52.544+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:52.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:52.545+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:53.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:53.066+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:53.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:53.066+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:53.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:53.066+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:53.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:53.067+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:53.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:53.005+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:53.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:53.005+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:53.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:53.005+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:53.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:53.006+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:53.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:53.573+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:53.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:53.573+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:53.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:53.573+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:53.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:53.574+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:54.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:54.082+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:54.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:54.082+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:54.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:54.082+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:54.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:54.082+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:54.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:53.980+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:54.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:53.980+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:54.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:53.980+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:54.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:53.981+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:54.978 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:54.615+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:54.978 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:54.615+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:54.978 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:54.615+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:54.978 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:54.616+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:55.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:54.976+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:55.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:54.976+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:55.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:54.976+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:55.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:54.978+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:55.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:55.107+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:55.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:55.107+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:55.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:55.107+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:55.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:55.108+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:56.001 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:35:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:35:55.671+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18179 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:35:56.001 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:55.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:56.001 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:55.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:56.001 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:55.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:56.002 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:55.610+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:56.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:56.149+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:56.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:56.149+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:56.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:56.149+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:56.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:56.150+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:56.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:56.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:56.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:56.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:56.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:56.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:56.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:56.001+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:57.028 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:56.621+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:57.029 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:56.621+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:57.029 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:56.621+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:57.029 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:56.622+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:57.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:57.027+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:57.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:57.028+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:57.028+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:57.028+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:57.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:57.175+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:57.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:57.175+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:57.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:57.175+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:57.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:57.176+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:58.064 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:57.671+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:58.064 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:57.671+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:58.064 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:57.671+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:58.064 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:57.672+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:58.063+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:58.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:58.063+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:58.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:58.063+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:58.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:58.063+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:58.201+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:58.201+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:58.201+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:58.202+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:35:59.044 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:58.630+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:59.044 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:58.630+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:59.044 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:58.630+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:35:59.044 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:58.631+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:59.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:59.043+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:59.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:59.043+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:59.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:59.043+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:35:59.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:35:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:35:59.044+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:35:59.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:59.206+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:59.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:59.206+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:59.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:59.206+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:35:59.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:35:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:35:59.207+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:00.030 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:59.658+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:00.031 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:59.658+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:00.031 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:59.658+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:00.031 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:35:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:35:59.659+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:00.292 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:00.232+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:00.292 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:00.232+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:00.292 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:00.232+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:00.292 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:00.233+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:00.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:00.029+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:00.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:00.029+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:00.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:00.029+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:00.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:00.030+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:01.033 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:36:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:36:00.673+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18268 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:36:01.033 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:00.631+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:01.033 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:00.631+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:01.033 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:00.631+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:01.033 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:00.632+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:01.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:01.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:01.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:01.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:01.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:01.032+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:01.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:01.234+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:01.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:01.234+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:01.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:01.234+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:01.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:01.235+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:02.079 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:01.616+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:02.080 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:01.616+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:02.080 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:01.616+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:02.080 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:01.616+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:02.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:02.078+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:02.078+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:02.078+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:02.079+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:02.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:02.265+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:02.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:02.265+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:02.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:02.265+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:02.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:02.265+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:03.041 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:02.648+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:03.041 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:02.648+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:03.041 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:02.648+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:03.041 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:02.649+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:03.284+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:03.284+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:03.284+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:03.285+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:03.040+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:03.040+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:03.040+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:03.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:03.041+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:03.999 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:03.656+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:03.999 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:03.656+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:03.999 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:03.656+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:03.999 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:03.657+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:04.273 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:03.998+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:04.273 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:03.998+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:04.273 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:03.998+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:04.273 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:03.998+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:04.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:04.272+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:04.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:04.272+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:04.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:04.272+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:04.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:04.273+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:05.026 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:04.688+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:05.026 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:04.688+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:05.026 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:04.688+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:05.026 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:04.689+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:05.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:05.025+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:05.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:05.025+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:05.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:05.025+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:05.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:05.026+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:05.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:05.225+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:05.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:05.225+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:05.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:05.225+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:05.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:05.225+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:06.070 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:36:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:36:05.675+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18358 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:36:06.070 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:05.713+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:06.070 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:05.713+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:06.070 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:05.713+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:06.071 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:05.713+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:06.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:06.268+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:06.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:06.268+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:06.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:06.268+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:06.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:06.268+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:06.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:06.069+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:06.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:06.069+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:06.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:06.069+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:06.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:06.070+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:06.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:06.666+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:06.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:06.666+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:06.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:06.666+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:06.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:06.667+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:07.270 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:07.036+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:07.271 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:07.036+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:07.271 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:07.036+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:07.271 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:07.037+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:07.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:07.269+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:07.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:07.269+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:07.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:07.269+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:07.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:07.270+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:08.039 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:07.658+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:08.039 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:07.658+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:08.039 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:07.658+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:08.039 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:07.659+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:08.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:08.232+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:08.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:08.232+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:08.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:08.232+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:08.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:08.233+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:08.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:08.038+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:08.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:08.038+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:08.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:08.038+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:08.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:08.039+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:09.015 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:08.625+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:09.016 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:08.625+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:09.016 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:08.625+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:09.016 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:08.625+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:09.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:09.221+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:09.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:09.221+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:09.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:09.221+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:09.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:09.222+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:09.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:09.015+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:09.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:09.015+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:09.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:09.015+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:09.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:09.015+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:09.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:09.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:09.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:09.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:09.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:09.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:09.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:09.596+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:10.251 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:09.967+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:10.252 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:09.967+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:10.252 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:09.967+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:10.252 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:09.968+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:10.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:10.250+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:10.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:10.250+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:10.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:10.250+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:10.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:10.251+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:10.933 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:36:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:36:10.677+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18447 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:36:10.933 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:10.932+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:10.933 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:10.932+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:10.934 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:10.638+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:10.934 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:10.638+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:10.934 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:10.638+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:10.934 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:10.638+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:11.299 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:10.932+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:11.299 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:10.933+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:11.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:11.298+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:11.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:11.298+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:11.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:11.298+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:11.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:11.299+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:11.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:11.630+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:11.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:11.630+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:11.630+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:11.631+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:12.292 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:11.948+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:12.292 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:11.948+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:12.292 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:11.948+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:12.292 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:11.949+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:12.580 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:12.291+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:12.580 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:12.291+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:12.580 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:12.291+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:12.580 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:12.292+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:12.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:12.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:12.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:12.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:12.647+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:13.300 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:12.984+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:13.300 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:12.984+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:13.300 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:12.984+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:13.300 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:12.984+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:13.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:13.299+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:13.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:13.299+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:13.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:13.299+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:40.800695+0000 (oldest deadline 2024-03-19T03:26:00.800695+0000) 2024-03-19T03:36:13.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:13.300+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:14.020 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:13.687+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:14.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:13.687+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:14.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:13.687+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:14.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:13.688+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:14.327 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:14.019+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:14.328 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:14.019+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:14.328 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:14.019+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:14.328 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:14.020+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:14.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:14.327+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:15.028 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:14.650+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:15.028 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:14.650+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:15.028 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:14.650+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:15.028 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:14.651+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:15.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:15.026+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:15.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:15.026+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:15.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:15.026+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:15.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:15.027+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:15.652 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:15.376+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:16.073 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:36:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:36:15.679+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18538 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:36:16.073 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:15.666+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:16.073 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:15.666+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:16.073 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:15.666+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:16.073 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:15.667+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:16.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:16.072+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:16.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:16.072+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:16.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:16.072+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:16.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:16.073+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:16.642 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:16.373+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:17.079 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:16.641+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:17.079 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:16.641+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:17.079 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:16.641+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:17.079 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:16.642+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:17.078+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:17.078+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:17.078+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:17.079+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:17.652 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:17.375+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:18.071 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:17.651+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:18.071 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:17.651+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:18.071 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:17.651+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:18.071 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:17.652+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:18.070+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:18.070+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:18.070+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:18.071+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:18.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:18.417+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:19.064 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:18.680+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:19.064 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:18.680+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:19.064 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:18.680+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:19.064 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:18.681+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:19.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:19.063+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:19.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:19.063+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:19.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:19.063+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:46.613413+0000 (oldest deadline 2024-03-19T03:26:06.613413+0000) 2024-03-19T03:36:19.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:19.064+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:19.706 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:19.420+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:20.030 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:19.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:20.030 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:19.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:20.030 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:19.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:20.030 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:19.706+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:20.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:20.029+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:20.683 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:20.424+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:20.993 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:36:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:36:20.681+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18627 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:36:20.993 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:20.722+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:20.993 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:20.722+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:20.993 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:20.722+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:20.993 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:20.723+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:21.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:20.992+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:21.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:21.448+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:22.043 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:21.770+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:22.043 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:21.770+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:22.043 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:21.770+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:22.043 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:21.771+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:22.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:22.042+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:22.755 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:22.438+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:23.068 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:22.754+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:23.068 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:22.754+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:23.068 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:22.754+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:23.068 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:22.755+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:23.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:23.067+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:23.747 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:23.471+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:24.062 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:23.746+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:24.062 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:23.746+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:24.062 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:23.746+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:24.062 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:23.747+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:24.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:24.061+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:24.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:24.439+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:25.036 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:24.701+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:25.036 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:24.702+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:25.036 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:24.702+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:25.036 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:24.702+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:25.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:25.035+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:25.707 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:36:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:36:25.683+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18717 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:36:25.707 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:25.446+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:26.070 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:25.731+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:26.070 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:25.731+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:26.070 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:25.731+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:26.070 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:25.731+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:26.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:26.069+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:26.757 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:26.397+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:27.085 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:26.756+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:27.085 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:26.756+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:27.085 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:26.756+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:25:52.749563+0000 (oldest deadline 2024-03-19T03:26:12.749563+0000) 2024-03-19T03:36:27.085 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:26.756+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:27.084+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:27.735 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:27.415+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:28.081 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:27.734+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:28.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:28.080+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:28.719 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:28.401+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:29.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:28.718+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:29.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:29.359+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:29.126+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:30.082 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:29.695+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:30.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:30.081+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:30.686 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:30.365+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:31.012 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:36:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:36:30.685+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18806 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:36:31.013 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:30.688+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:31.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:31.094+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:31.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:31.352+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:32.084 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:31.709+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:32.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:32.083+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:32.741 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:32.394+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:33.062 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:32.741+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:33.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:33.061+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:33.754 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:33.436+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:34.056 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:33.753+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:34.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:34.055+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:34.729 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:34.447+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:35.016 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:34.729+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:35.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:35.015+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:35.712 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:36:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:36:35.687+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18897 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:36:35.712 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:35.456+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:35.713 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:35.456+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:35.713 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:35.456+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:35.713 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:35.457+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:36.007 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:35.715+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:36.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:36.007+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:36.700+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:36.498+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:36.498+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:36.498+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:36.499+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:37.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:37.032+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:37.667 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:37.500+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:37.667 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:37.500+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:37.667 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:37.500+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:37.667 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:37.500+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:38.068 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:37.666+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:38.067+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:38.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:38.634+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:38.498+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:38.498+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:38.498+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:38.500+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:39.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:39.033+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:39.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:39.667+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:39.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:39.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:39.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:39.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:39.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:39.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:39.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:39.493+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:40.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:40.036+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:40.815 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:36:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:36:40.689+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18986 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:36:40.815 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:40.652+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:40.815 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:40.536+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:40.815 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:40.536+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:40.816 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:40.536+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:40.816 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:40.537+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:41.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:41.051+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:41.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:41.638+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:41.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:41.556+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:41.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:41.556+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:41.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:41.556+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:41.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:41.557+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:42.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:42.018+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:42.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:42.018+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:42.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:42.018+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:42.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:42.018+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:42.981 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:42.615+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:42.981 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:42.604+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:42.981 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:42.604+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:42.981 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:42.604+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:42.981 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:42.605+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:43.300 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:42.980+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:43.300 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:42.980+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:43.300 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:42.980+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:43.300 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:42.980+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:43.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:43.608+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:43.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:43.648+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:43.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:43.648+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:43.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:43.648+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:43.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:43.648+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:43.975+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:43.975+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:43.975+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:43.976+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:44.567+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:44.629+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:44.629+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:44.629+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:44.630+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:44.992+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:44.992+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:45.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:44.992+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:45.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:44.993+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:46.013 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:36:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:36:45.691+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19076 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:36:46.013 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:45.614+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:46.013 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:45.666+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:46.013 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:45.666+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:46.014 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:45.666+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:46.014 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:45.667+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:46.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:46.009+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:46.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:46.009+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:46.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:46.009+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:46.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:46.010+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:46.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:46.605+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:47.049 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:46.686+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:47.049 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:46.686+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:47.050 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:46.686+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:47.050 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:46.687+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:47.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:47.048+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:47.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:47.048+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:47.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:47.048+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:47.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:47.049+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:47.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:47.560+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:47.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:47.708+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:47.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:47.708+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:47.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:47.708+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:47.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:47.709+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:48.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:48.062+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:48.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:48.062+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:48.062+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:48.063+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:49.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:48.607+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:48.709+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:49.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:48.709+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:49.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:48.709+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:49.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:48.710+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:49.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:49.100+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:49.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:49.100+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:49.100+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:49.101+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:50.055 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:49.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:50.056 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:49.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:50.056 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:49.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:50.056 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:49.643+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:50.056 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:49.724+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:50.056 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:49.724+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:50.056 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:49.724+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:50.056 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:49.725+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:50.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:50.054+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:50.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:50.054+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:50.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:50.054+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:50.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:50.055+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:51.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:36:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:36:50.693+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19166 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:36:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:50.661+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:50.661+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:50.661+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:50.662+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:51.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:50.682+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:51.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:50.682+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:51.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:50.682+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:51.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:50.683+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:51.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:51.097+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:51.097+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:51.097+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:51.098+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:52.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:51.652+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:51.652+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:51.652+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:51.653+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:51.667+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:51.667+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:51.667+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:51.668+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:52.516 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:52.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:52.516 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:52.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:52.516 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:52.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:52.516 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:52.132+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:52.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:52.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:52.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:52.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:52.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:52.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:52.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:52.642+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:52.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:52.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:52.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:52.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:52.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:52.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:52.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:52.604+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:53.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:53.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:53.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:53.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:53.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:53.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:53.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:53.106+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:54.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:53.597+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:54.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:53.597+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:54.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:53.597+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:54.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:53.598+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:54.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:53.597+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:54.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:53.597+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:54.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:53.597+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:54.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:53.598+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:54.553 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:54.093+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:54.553 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:54.093+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:54.553 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:54.093+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:54.553 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:54.094+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:54.562+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:54.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:54.562+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:54.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:54.562+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:54.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:54.562+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:54.552+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:54.552+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:54.552+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:54.553+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:55.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:55.045+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:55.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:55.045+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:55.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:55.045+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:55.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:55.046+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:55.821 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:36:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:36:55.695+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19256 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:36:55.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:55.523+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:55.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:55.523+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:55.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:55.523+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:55.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:55.524+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:55.822 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:55.577+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:55.822 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:55.577+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:55.822 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:55.577+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:55.822 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:55.578+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:56.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:56.035+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:56.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:56.035+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:56.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:56.035+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:56.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:56.036+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:56.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:56.558+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:56.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:56.558+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:56.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:56.558+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:56.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:56.558+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:56.557+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:56.557+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:56.557+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:56.558+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:57.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:57.078+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:57.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:57.078+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:57.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:57.078+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:57.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:57.079+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:57.546+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:57.546+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:57.546+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:57.547+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:57.583+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:57.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:57.583+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:57.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:57.583+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:57.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:57.583+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:58.054+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:58.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:58.054+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:58.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:58.054+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:58.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:58.055+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:58.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:58.578+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:58.578+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:58.578+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:58.578+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:36:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:58.599+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:58.599+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:58.599+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:58.599+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:59.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:59.038+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:59.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:59.038+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:59.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:59.038+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:36:59.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:59.039+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:59.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:59.587+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:59.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:59.587+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:59.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:59.587+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:36:59.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:36:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:36:59.587+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:36:59.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:59.602+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:59.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:59.602+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:59.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:59.602+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:36:59.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:36:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:36:59.603+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:00.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:59.998+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:00.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:59.998+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:00.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:36:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:59.998+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:00.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:36:59.999+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:00.983 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:37:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:37:00.697+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19345 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:37:00.984 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:00.617+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:00.984 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:00.617+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:00.984 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:00.617+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:00.984 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:00.617+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:00.984 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:00.618+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:00.984 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:00.618+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:00.984 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:00.618+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:00.984 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:00.619+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:00.982+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:00.982+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:00.982+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:01.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:00.984+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:02.032 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:01.663+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:02.032 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:01.663+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:02.032 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:01.663+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:02.032 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:01.664+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:02.032 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:01.630+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:02.033 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:01.630+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:02.033 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:01.630+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:02.033 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:01.630+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:02.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:02.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:02.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:02.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:02.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:02.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:02.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:02.032+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:03.058 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:02.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:03.058 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:02.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:03.058 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:02.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:03.058 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:02.604+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:03.058 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:02.655+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:03.058 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:02.655+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:03.058 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:02.655+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:03.059 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:02.656+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:03.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:03.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:03.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:03.058+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:04.074 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:03.668+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:04.074 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:03.668+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:04.074 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:03.668+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:04.074 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:03.669+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:04.074 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:03.607+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:04.074 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:03.607+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:04.074 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:03.607+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:04.074 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:03.608+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:04.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:04.073+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:04.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:04.073+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:04.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:04.073+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:04.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:04.074+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:05.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:04.672+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:05.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:04.672+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:05.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:04.672+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:05.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:04.673+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:04.613+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:04.613+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:04.613+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:04.614+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:05.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:05.088+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:05.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:05.088+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:05.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:05.088+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:05.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:05.089+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:06.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:37:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:37:05.700+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19436 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:37:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:05.704+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:05.704+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:05.704+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:05.706+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:05.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:05.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:06.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:05.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:06.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:05.604+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:06.557 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:06.097+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:06.558 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:06.097+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:06.558 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:06.097+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:06.558 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:06.098+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:06.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:06.724+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:06.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:06.724+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:06.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:06.724+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:06.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:06.725+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:06.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:06.557+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:06.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:06.557+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:06.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:06.557+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:06.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:06.558+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:07.532 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:07.092+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:07.532 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:07.092+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:07.532 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:07.092+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:07.532 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:07.093+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:07.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:07.722+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:07.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:07.722+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:07.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:07.722+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:07.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:07.722+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:07.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:07.532+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:07.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:07.532+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:07.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:07.532+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:07.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:07.532+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:08.114+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:08.114+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:08.114+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:08.114+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:08.685 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:08.526+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:08.685 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:08.526+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:08.686 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:08.526+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:08.686 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:08.527+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:09.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:08.685+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:09.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:08.685+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:09.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:08.685+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:09.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:08.686+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:09.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:09.129+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:09.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:09.129+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:09.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:09.129+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:09.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:09.130+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:09.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:09.550+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:09.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:09.550+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:09.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:09.550+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:09.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:09.551+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:09.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:09.708+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:09.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:09.708+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:09.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:09.708+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:09.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:09.709+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:10.517 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:10.093+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:10.518 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:10.093+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:10.518 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:10.093+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:10.518 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:10.093+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:10.825 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:37:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:37:10.701+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19527 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:37:10.826 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:10.738+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:10.826 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:10.738+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:10.826 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:10.738+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:10.826 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:10.739+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:10.826 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:10.516+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:10.826 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:10.516+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:10.826 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:10.516+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:10.826 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:10.517+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:11.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:11.111+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:11.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:11.111+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:11.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:11.111+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:11.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:11.111+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:11.783 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:11.522+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:11.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:11.522+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:11.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:11.522+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:11.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:11.523+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:12.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:11.783+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:12.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:11.783+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:12.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:11.783+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:12.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:11.784+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:12.556 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:12.153+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:12.556 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:12.153+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:12.556 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:12.153+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:12.556 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:12.153+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:12.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:12.555+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:12.555+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:12.555+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:12.556+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:12.788+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:12.788+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:12.788+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:12.789+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:13.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:13.158+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:13.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:13.158+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:13.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:13.158+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:13.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:13.159+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:14.020 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:13.832+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:14.020 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:13.832+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:14.020 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:13.832+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:14.020 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:13.833+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:14.020 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:13.587+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:14.020 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:13.587+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:14.021 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:13.587+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:14.021 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:13.588+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:14.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:14.175+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:14.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:14.175+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:14.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:14.175+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:14.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:14.176+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:14.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:14.583+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:14.899 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:14.583+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:14.899 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:14.583+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:14.899 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:14.584+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:15.188 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:14.875+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:15.189 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:14.875+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:15.189 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:14.875+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:15.189 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:14.876+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:15.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:15.188+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:15.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:15.188+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:15.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:15.188+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:15.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:15.188+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:15.827 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:37:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:37:15.703+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19618 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:37:15.828 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:15.550+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:15.828 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:15.550+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:15.828 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:15.550+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:15.828 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:15.551+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:16.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:15.836+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:16.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:15.836+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:15.836+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:15.836+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:16.579 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:16.168+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:16.580 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:16.168+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:16.580 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:16.168+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:16.580 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:16.169+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:16.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:16.809+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:16.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:16.809+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:16.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:16.809+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:16.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:16.809+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:16.578+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:16.578+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:16.578+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:16.579+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:17.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:17.182+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:17.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:17.182+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:17.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:17.182+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:17.183+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:18.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:17.590+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:17.590+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:17.590+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:17.591+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:18.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:17.806+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:18.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:17.806+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:18.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:17.806+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:18.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:17.807+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:18.542 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:18.197+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:18.543 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:18.197+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:18.543 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:18.197+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:18.543 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:18.198+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:18.831 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:18.542+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:18.831 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:18.542+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:18.831 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:18.542+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:18.831 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:18.542+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:19.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:18.830+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:19.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:18.830+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:19.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:18.830+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:19.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:18.830+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:19.548 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:19.202+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:19.548 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:19.202+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:19.548 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:19.202+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:19.548 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:19.203+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:19.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:19.547+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:19.802 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:19.547+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:19.802 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:19.547+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:19.802 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:19.548+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:20.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:19.801+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:20.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:19.801+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:20.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:19.801+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:20.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:19.802+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:20.581 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:20.154+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:20.581 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:20.154+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:20.582 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:20.154+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:20.582 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:20.155+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:20.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:37:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:37:20.705+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19708 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:37:20.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:20.766+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:20.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:20.766+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:20.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:20.766+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:20.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:20.767+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:20.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:20.580+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:20.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:20.580+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:20.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:20.580+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:20.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:20.581+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:21.573 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:21.194+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:21.573 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:21.194+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:21.573 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:21.194+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:21.573 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:21.195+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:21.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:21.724+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:21.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:21.724+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:21.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:21.724+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:21.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:21.725+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:21.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:21.572+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:21.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:21.572+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:21.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:21.572+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:21.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:21.572+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:22.578 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:22.172+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:22.578 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:22.172+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:22.578 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:22.172+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:22.578 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:22.173+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:22.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:22.742+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:22.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:22.742+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:22.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:22.742+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:22.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:22.743+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:22.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:22.577+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:22.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:22.577+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:22.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:22.577+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:22.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:22.578+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:23.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:23.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:23.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:23.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:23.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:23.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:23.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:23.133+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:24.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:23.619+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:24.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:23.619+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:24.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:23.619+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:24.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:23.619+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:24.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:23.739+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:24.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:23.739+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:24.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:23.739+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:24.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:23.740+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:24.581 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:24.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:24.581 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:24.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:24.581 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:24.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:24.581 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:24.137+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:24.754+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:24.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:24.754+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:24.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:24.754+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:24.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:24.755+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:24.841 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:24.580+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:24.841 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:24.580+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:24.841 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:24.580+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:24.841 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:24.581+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:25.556 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:25.090+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:25.556 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:25.090+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:25.556 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:25.090+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:25.556 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:25.091+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:25.833 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:37:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:37:25.707+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19799 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:37:25.833 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:25.772+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:25.834 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:25.772+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:25.834 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:25.772+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:25.834 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:25.772+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:25.834 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:25.555+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:25.834 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:25.555+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:25.834 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:25.555+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:25.834 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:25.556+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:26.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:26.060+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:26.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:26.060+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:26.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:26.060+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:26.061+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:26.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:26.523+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:26.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:26.523+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:26.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:26.523+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:26.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:26.524+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:27.032 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:26.735+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:27.032 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:26.735+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:27.032 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:26.735+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:27.033 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:26.735+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:27.031+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:27.031+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:27.031+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:27.032+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:27.841 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:27.779+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:27.841 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:27.779+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:27.841 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:27.779+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:27.841 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:27.780+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:27.842 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:27.558+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:27.842 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:27.558+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:27.842 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:27.558+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:27.842 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:27.559+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:28.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:28.073+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:28.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:28.073+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:28.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:28.073+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:28.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:28.073+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:28.815 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:28.548+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:28.815 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:28.548+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:28.815 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:28.548+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:28.815 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:28.548+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:29.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:28.814+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:29.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:28.814+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:29.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:28.814+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:29.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:28.815+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:29.381 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:29.121+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:29.381 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:29.121+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:29.381 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:29.121+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:29.381 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:29.121+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:29.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:29.809+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:29.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:29.809+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:29.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:29.809+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:29.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:29.810+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:29.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:29.574+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:29.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:29.574+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:29.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:29.574+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:29.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:29.575+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:30.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:30.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:30.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:30.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:30.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:30.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:30.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:30.142+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:31.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:37:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:37:30.709+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19890 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:37:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:30.601+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:30.601+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:30.601+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:30.602+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:31.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:30.795+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:31.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:30.795+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:31.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:30.795+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:31.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:30.795+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:31.553 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:31.178+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:31.553 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:31.178+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:31.553 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:31.178+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:31.553 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:31.179+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:31.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:31.552+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:31.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:31.552+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:31.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:31.552+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:31.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:31.553+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:31.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:31.766+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:31.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:31.766+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:31.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:31.766+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:31.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:31.766+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:32.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:32.180+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:32.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:32.180+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:32.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:32.180+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:32.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:32.181+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:32.770 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:32.504+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:32.770 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:32.504+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:32.770 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:32.504+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:32.770 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:32.505+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:33.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:32.769+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:33.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:32.769+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:33.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:32.769+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:33.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:32.770+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:33.502 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:33.197+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:33.503 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:33.197+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:33.503 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:33.197+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:33.503 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:33.198+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:33.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:33.747+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:33.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:33.747+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:33.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:33.747+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:33.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:33.747+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:33.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:33.501+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:33.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:33.501+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:33.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:33.501+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:33.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:33.502+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:34.506 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:34.203+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:34.506 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:34.203+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:34.506 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:34.203+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:34.506 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:34.204+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:34.767 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:34.505+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:34.768 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:34.505+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:34.768 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:34.505+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:34.768 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:34.505+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:35.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:34.766+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:35.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:34.766+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:35.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:34.766+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:35.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:34.767+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:35.516 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:35.218+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:35.516 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:35.218+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:35.516 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:35.218+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:35.517 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:35.219+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:35.836 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:37:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:37:35.711+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19981 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:37:35.836 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:35.515+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:35.836 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:35.515+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:35.836 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:35.515+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:35.836 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:35.516+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:35.836 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:35.758+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:35.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:35.758+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:35.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:35.758+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:35.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:35.759+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:36.555 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:36.173+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:36.556 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:36.173+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:36.556 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:36.173+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:36.556 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:36.174+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:36.735+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:36.735+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:36.735+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:36.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:36.735+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:36.555+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:36.555+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:36.555+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:36.556+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:37.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:37.203+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:37.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:37.203+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:37.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:37.203+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:37.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:37.203+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:37.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:37.545+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:37.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:37.545+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:37.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:37.545+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:37.546+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:37.756+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:37.756+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:37.756+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:37.757+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:38.553 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:38.237+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:38.553 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:38.237+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:38.553 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:38.237+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:38.553 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:38.238+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:38.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:38.552+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:38.552+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:38.552+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:38.553+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:38.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:38.800+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:38.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:38.800+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:38.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:38.800+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:38.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:38.801+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:39.525 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:39.262+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:39.525 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:39.262+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:39.525 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:39.262+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:39.525 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:39.263+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:39.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:39.761+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:39.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:39.761+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:39.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:39.761+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:39.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:39.761+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:39.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:39.524+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:39.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:39.524+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:39.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:39.524+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:39.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:39.525+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:40.531 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:40.244+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:40.531 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:40.244+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:40.531 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:40.244+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:40.531 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:40.244+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:40.798 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:37:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:37:40.712+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20071 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:37:40.798 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:40.530+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:40.798 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:40.530+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:40.798 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:40.530+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:40.798 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:40.531+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:41.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:40.797+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:41.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:40.797+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:41.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:40.797+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:41.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:40.798+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:41.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:41.511+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:41.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:41.511+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:41.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:41.511+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:41.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:41.512+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:41.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:41.264+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:41.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:41.264+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:41.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:41.264+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:41.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:41.265+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:42.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:41.834+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:42.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:41.834+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:42.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:41.834+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:42.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:41.835+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:42.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:42.498+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:42.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:42.498+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:42.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:42.498+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:42.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:42.499+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:42.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:42.314+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:42.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:42.314+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:42.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:42.314+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:42.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:42.315+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:43.287 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:42.842+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:43.287 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:42.842+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:43.287 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:42.842+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:43.288 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:42.843+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:43.288 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:43.286+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:43.288 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:43.286+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:43.288 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:43.286+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:43.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:43.287+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:43.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:43.456+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:43.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:43.456+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:43.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:43.456+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:43.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:43.457+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:44.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:43.835+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:44.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:43.835+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:44.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:43.835+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:44.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:43.835+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:44.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:44.472+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:44.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:44.472+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:44.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:44.472+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:44.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:44.473+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:44.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:44.244+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:44.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:44.244+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:44.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:44.244+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:44.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:44.244+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:44.807+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:44.807+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:45.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:44.807+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:45.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:44.808+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:45.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:45.277+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:45.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:45.277+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:45.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:45.277+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:45.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:45.278+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:45.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:45.518+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:45.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:45.518+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:45.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:45.518+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:45.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:45.519+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:46.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:37:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:37:45.715+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20162 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:37:46.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:45.847+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:46.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:45.847+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:46.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:45.847+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:46.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:45.849+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:46.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:46.231+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:46.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:46.231+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:46.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:46.231+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:46.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:46.232+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:46.821 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:46.546+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:46.821 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:46.546+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:46.822 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:46.546+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:46.822 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:46.547+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:47.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:46.820+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:47.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:46.820+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:47.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:46.820+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:47.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:46.821+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:47.536 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:47.219+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:47.536 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:47.219+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:47.536 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:47.219+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:47.536 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:47.220+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:47.804 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:47.535+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:47.805 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:47.535+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:47.805 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:47.535+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:47.805 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:47.536+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:48.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:47.803+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:48.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:47.803+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:48.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:47.803+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:48.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:47.804+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:48.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:48.175+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:48.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:48.175+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:48.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:48.175+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:48.524 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:48.176+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:48.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:48.522+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:48.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:48.522+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:48.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:48.522+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:48.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:48.523+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:49.128 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:48.840+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:49.128 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:48.840+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:49.128 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:48.840+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:49.129 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:48.841+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:49.567 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:49.127+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:49.567 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:49.127+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:49.567 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:49.127+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:49.567 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:49.128+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:49.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:49.566+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:49.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:49.566+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:49.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:49.566+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:49.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:49.567+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:50.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:49.836+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:50.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:49.836+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:50.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:49.836+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:50.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:49.837+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:50.578 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:50.172+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:50.578 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:50.172+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:50.578 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:50.172+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:50.578 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:50.172+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:50.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:37:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:37:50.717+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20253 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:37:50.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:50.817+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:50.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:50.817+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:50.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:50.817+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:50.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:50.818+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:50.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:50.577+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:50.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:50.577+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:50.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:50.577+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:50.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:50.577+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:51.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:51.134+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:51.134+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:51.134+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:51.135+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:52.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:51.589+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:51.589+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:51.589+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:51.589+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:51.775+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:51.775+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:51.775+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:51.775+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:52.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:52.171+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:52.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:52.171+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:52.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:52.171+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:52.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:52.172+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:53.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:52.605+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:53.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:52.605+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:53.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:52.605+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:53.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:52.606+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:53.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:52.812+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:53.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:52.812+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:53.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:52.812+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:53.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:52.812+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:53.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:53.128+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:53.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:53.128+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:53.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:53.128+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:53.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:53.129+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:53.956 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:53.614+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:53.956 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:53.614+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:53.956 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:53.614+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:53.956 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:53.614+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:53.956 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:53.812+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:53.956 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:53.812+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:53.956 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:53.812+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:53.956 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:53.813+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:54.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:54.148+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:54.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:54.148+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:54.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:54.148+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:54.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:54.149+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:55.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:54.634+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:55.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:54.634+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:55.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:54.634+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:55.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:54.635+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:55.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:54.800+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:55.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:54.800+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:55.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:54.800+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:55.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:54.801+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:55.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:55.109+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:55.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:55.109+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:55.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:55.109+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:55.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:55.110+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:56.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:37:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:37:55.720+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20343 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:37:56.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:55.773+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:56.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:55.773+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:56.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:55.773+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:56.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:55.774+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:56.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:55.670+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:56.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:55.670+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:56.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:55.670+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:56.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:55.671+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:56.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:56.117+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:56.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:56.117+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:56.117+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:56.118+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:57.029 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:56.786+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:57.029 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:56.786+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:57.029 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:56.786+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:57.029 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:56.786+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:57.029 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:56.696+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:57.029 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:56.696+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:57.029 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:56.696+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:57.029 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:56.697+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:57.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:57.142+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:57.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:57.142+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:57.142+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:57.143+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:58.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:57.739+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:58.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:57.739+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:58.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:57.739+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:58.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:57.740+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:58.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:57.771+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:58.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:57.771+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:58.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:57.771+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:58.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:57.772+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:58.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:58.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:58.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:58.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:58.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:58.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:58.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:58.142+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:59.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:58.731+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:59.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:58.731+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:59.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:58.731+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:37:59.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:58.731+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:37:59.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:58.748+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:59.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:58.748+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:59.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:58.748+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:37:59.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:58.748+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:37:59.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:59.128+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:59.128+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:59.128+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:37:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:37:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:37:59.129+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:00.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:59.714+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:00.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:59.714+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:00.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:59.714+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:00.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:37:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:37:59.715+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:59.795+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:59.795+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:59.795+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:37:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:37:59.795+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:00.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:00.131+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:00.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:00.131+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:00.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:00.131+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:00.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:00.132+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:01.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:38:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:38:00.722+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20433 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:38:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:00.711+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:00.711+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:00.711+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:00.712+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:01.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:00.840+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:01.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:00.840+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:01.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:00.840+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:01.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:00.841+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:01.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:01.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:01.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:01.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:01.167+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:02.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:01.846+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:02.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:01.846+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:02.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:01.846+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:02.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:01.847+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:01.743+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:01.743+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:01.743+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:01.744+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:02.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:02.185+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:02.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:02.185+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:02.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:02.185+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:02.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:02.186+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:03.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:02.812+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:03.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:02.812+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:03.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:02.812+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:03.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:02.813+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:02.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:02.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:02.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:02.745+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:03.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:03.138+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:03.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:03.138+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:03.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:03.138+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:03.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:03.138+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:04.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:03.823+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:04.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:03.823+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:04.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:03.823+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:04.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:03.824+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:04.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:03.770+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:04.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:03.770+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:04.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:03.770+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:04.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:03.771+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:04.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:04.127+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:04.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:04.127+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:04.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:04.127+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:04.127+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:05.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:04.825+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:05.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:04.825+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:05.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:04.825+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:05.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:04.826+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:04.800+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:04.800+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:04.800+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:04.801+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:05.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:05.111+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:05.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:05.111+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:05.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:05.111+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:05.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:05.111+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:06.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:38:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:38:05.724+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20524 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:38:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:05.860+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:05.860+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:05.860+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:05.860+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:05.819+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:05.819+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:05.819+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:05.819+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:06.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:06.090+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:06.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:06.090+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:06.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:06.090+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:06.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:06.091+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:07.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:06.898+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:07.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:06.899+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:07.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:06.899+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:07.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:06.899+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:06.772+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:06.772+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:06.772+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:06.773+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:07.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:07.092+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:07.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:07.092+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:07.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:07.092+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:07.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:07.093+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:08.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:07.893+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:08.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:07.893+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:08.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:07.893+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:08.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:07.894+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:07.735+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:07.735+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:07.735+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:07.736+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:08.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:08.117+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:08.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:08.117+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:08.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:08.117+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:08.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:08.118+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:09.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:08.908+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:09.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:08.908+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:09.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:08.908+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:09.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:08.909+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:08.707+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:08.707+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:08.707+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:08.708+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:09.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:09.096+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:09.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:09.096+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:09.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:09.096+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:09.097+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:09.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:09.710+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:09.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:09.710+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:09.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:09.710+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:09.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:09.711+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:10.176 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:09.915+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:10.176 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:09.915+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:10.176 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:09.915+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:10.176 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:09.916+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:10.176 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:10.069+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:10.176 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:10.069+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:10.176 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:10.069+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:10.177 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:10.070+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:11.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:38:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:38:10.726+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20614 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:38:11.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:10.966+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:11.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:10.966+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:11.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:10.966+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:11.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:10.967+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:11.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:10.748+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:11.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:10.748+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:11.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:10.748+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:11.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:10.749+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:11.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:11.088+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:11.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:11.088+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:11.088+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:11.089+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:12.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:11.740+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:11.740+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:11.740+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:11.740+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:12.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:11.943+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:12.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:11.943+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:12.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:11.943+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:12.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:11.944+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:12.388 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:12.125+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:12.388 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:12.125+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:12.388 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:12.125+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:12.388 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:12.126+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:13.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:12.894+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:13.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:12.894+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:13.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:12.894+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:12.895+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:12.739+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:12.739+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:12.739+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:12.740+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:13.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:13.126+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:13.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:13.126+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:13.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:13.126+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:13.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:13.127+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:13.869 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:13.784+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:13.869 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:13.784+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:13.869 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:13.784+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:13.869 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:13.785+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:14.148 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:13.868+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:14.148 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:13.868+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:14.148 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:13.868+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:14.149 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:13.869+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:14.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:14.147+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:14.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:14.147+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:14.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:14.147+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:14.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:14.148+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:15.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:14.761+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:14.761+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:14.761+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:14.762+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:15.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:14.909+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:15.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:14.909+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:15.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:14.909+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:15.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:14.909+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:15.460 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:15.115+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:15.460 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:15.115+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:15.460 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:15.115+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:15.460 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:15.116+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:16.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:38:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:38:15.728+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20703 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:38:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:15.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:15.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:15.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:15.795+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:15.950+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:15.950+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:15.950+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:15.951+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:16.103+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:16.103+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:16.103+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:16.103+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:17.080 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:16.914+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:17.081 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:16.914+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:17.081 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:16.914+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:17.081 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:16.915+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:17.081 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:16.822+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:17.081 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:16.822+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:17.081 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:16.822+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:17.081 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:16.823+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:17.079+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:17.079+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:17.079+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:17.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:17.080+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:17.831+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:17.831+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:17.831+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:17.832+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:18.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:17.868+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:18.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:17.868+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:18.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:17.868+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:18.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:17.869+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:18.532 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:18.111+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:18.533 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:18.111+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:18.533 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:18.111+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:18.533 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:18.112+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:19.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:18.847+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:19.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:18.847+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:19.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:18.847+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:19.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:18.848+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:18.818+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:18.818+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:18.818+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:18.819+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:19.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:19.131+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:19.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:19.131+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:19.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:19.131+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:19.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:19.131+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:20.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:19.883+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:20.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:19.883+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:20.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:19.883+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:20.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:19.883+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:20.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:19.782+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:20.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:19.782+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:20.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:19.782+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:20.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:19.782+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:20.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:20.165+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:20.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:20.165+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:20.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:20.165+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:20.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:20.165+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:21.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:38:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:38:20.730+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20786 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:38:21.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:20.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:21.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:20.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:21.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:20.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:21.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:20.770+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:21.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:20.894+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:21.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:20.894+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:21.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:20.894+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:21.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:20.895+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:21.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:21.212+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:21.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:21.212+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:21.212+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:21.212+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:21.883 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:21.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:21.883 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:21.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:21.883 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:21.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:21.883 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:21.742+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:22.233 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:21.883+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:22.233 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:21.883+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:22.233 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:21.883+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:22.233 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:21.884+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:22.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:22.233+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:22.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:22.233+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:22.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:22.233+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:22.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:22.234+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:22.775+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:22.775+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:22.775+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:22.776+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:23.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:22.893+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:23.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:22.893+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:23.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:22.893+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:23.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:22.894+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:23.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:23.258+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:23.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:23.258+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:23.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:23.258+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:23.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:23.259+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:23.868 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:23.807+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:23.868 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:23.807+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:23.868 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:23.807+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:23.868 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:23.808+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:24.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:23.868+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:24.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:23.868+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:24.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:23.868+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:24.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:23.869+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:24.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:24.218+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:24.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:24.218+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:24.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:24.218+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:24.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:24.219+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:25.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:24.844+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:25.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:24.844+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:25.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:24.844+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:25.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:24.844+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:25.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:24.792+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:25.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:24.792+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:25.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:24.792+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:25.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:24.792+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:25.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:25.198+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:25.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:25.198+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:25.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:25.198+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:25.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:25.199+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:26.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:38:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:38:25.734+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20871 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:38:26.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:25.890+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:26.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:25.890+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:26.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:25.890+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:26.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:25.891+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:26.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:25.774+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:26.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:25.774+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:26.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:25.774+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:26.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:25.775+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:26.158+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:26.158+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:26.158+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:26.158+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:27.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:26.914+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:27.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:26.914+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:27.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:26.914+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:27.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:26.915+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:27.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:26.766+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:27.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:26.766+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:27.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:26.766+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:27.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:26.766+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:27.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:27.168+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:27.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:27.168+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:27.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:27.168+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:27.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:27.169+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:28.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:27.930+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:28.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:27.930+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:28.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:27.930+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:28.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:27.930+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:28.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:27.764+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:28.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:27.764+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:28.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:27.764+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:28.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:27.764+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:28.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:28.211+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:28.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:28.211+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:28.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:28.211+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:28.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:28.211+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:29.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:28.918+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:29.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:28.918+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:29.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:28.918+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:29.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:28.918+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:28.715+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:28.715+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:28.715+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:28.716+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:29.191+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:29.191+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:29.191+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:29.191+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:29.957 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:29.703+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:29.957 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:29.703+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:29.958 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:29.703+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:29.958 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:29.703+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:30.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:29.956+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:30.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:29.956+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:29.956+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:29.957+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:30.176+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:30.176+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:30.176+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:30.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:30.177+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:31.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:38:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:38:30.735+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20951 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:38:31.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:30.934+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:31.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:30.934+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:31.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:30.934+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:31.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:30.934+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:30.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:30.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:30.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:31.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:30.744+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:31.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:31.183+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:31.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:31.183+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:31.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:31.183+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:31.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:31.184+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:32.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:31.727+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:32.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:31.727+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:31.727+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:31.728+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:32.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:31.946+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:32.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:31.946+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:32.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:31.946+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:32.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:31.946+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:32.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:32.143+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:32.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:32.143+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:32.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:32.143+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:32.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:32.144+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:33.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:32.985+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:33.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:32.985+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:33.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:32.986+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:33.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:32.986+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:32.767+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:32.767+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:32.767+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:32.767+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:33.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:33.160+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:33.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:33.160+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:33.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:33.160+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:33.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:33.161+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:33.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:33.797+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:33.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:33.797+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:33.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:33.797+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:33.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:33.797+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:34.160 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:33.954+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:34.161 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:33.954+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:34.161 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:33.954+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:34.161 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:33.956+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:34.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:34.159+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:34.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:34.159+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:34.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:34.159+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:34.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:34.160+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:35.191 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:34.845+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:35.191 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:34.845+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:35.191 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:34.845+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:35.191 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:34.846+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:35.191 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:34.964+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:35.192 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:34.964+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:35.192 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:34.964+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:35.192 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:34.965+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:35.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:35.190+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:35.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:35.190+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:35.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:35.190+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:35.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:35.191+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:35.994 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:38:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:38:35.737+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21036 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:38:35.994 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:35.860+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:35.995 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:35.860+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:35.995 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:35.860+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:35.995 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:35.862+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:35.993+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:35.993+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:35.993+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:35.994+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:36.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:36.180+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:36.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:36.180+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:36.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:36.180+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:36.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:36.181+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:37.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:36.900+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:37.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:36.900+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:37.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:36.900+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:37.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:36.901+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:37.161 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:36.972+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:37.161 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:36.972+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:37.161 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:36.972+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:37.161 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:36.973+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:37.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:37.159+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:37.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:37.159+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:37.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:37.159+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:37.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:37.160+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:38.159 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:37.928+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:38.159 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:37.928+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:38.159 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:37.928+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:38.159 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:37.929+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:38.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:37.865+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:38.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:37.865+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:38.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:37.865+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:38.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:37.866+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:38.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:38.158+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:38.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:38.158+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:38.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:38.158+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:38.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:38.159+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:39.206 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:38.902+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:39.206 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:38.902+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:39.206 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:38.902+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:39.206 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:38.902+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:39.207 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:38.862+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:39.207 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:38.862+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:39.207 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:38.862+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:39.207 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:38.863+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:39.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:39.205+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:39.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:39.205+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:39.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:39.205+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:39.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:39.206+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:40.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:39.929+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:40.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:39.929+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:40.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:39.929+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:40.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:39.929+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:39.819+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:39.819+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:39.819+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:39.819+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:40.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:40.214+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:40.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:40.214+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:40.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:40.214+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:40.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:40.214+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:41.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:38:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:38:40.740+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21120 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:38:41.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:40.929+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:41.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:40.929+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:41.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:40.929+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:41.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:40.929+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:41.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:40.788+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:41.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:40.788+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:41.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:40.788+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:41.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:40.789+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:41.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:41.262+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:41.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:41.262+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:41.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:41.262+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:41.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:41.263+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:42.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:41.932+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:42.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:41.932+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:42.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:41.932+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:42.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:41.933+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:42.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:41.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:42.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:41.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:42.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:41.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:42.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:41.828+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:42.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:42.279+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:42.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:42.279+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:42.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:42.279+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:42.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:42.280+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:43.240 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:42.970+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:43.240 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:42.970+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:43.240 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:42.970+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:43.241 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:42.971+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:43.241 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:42.871+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:43.241 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:42.871+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:43.241 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:42.871+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:43.241 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:42.872+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:43.241 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:43.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:43.241 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:43.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:43.241 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:43.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:43.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:43.240+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:44.262 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:43.862+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:44.262 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:43.862+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:44.262 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:43.862+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:44.262 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:43.862+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:44.262 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:43.934+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:44.262 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:43.934+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:44.262 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:43.934+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:44.263 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:43.934+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:44.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:44.261+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:44.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:44.261+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:44.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:44.261+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:44.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:44.261+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:45.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:44.823+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:45.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:44.823+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:45.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:44.823+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:45.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:44.824+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:45.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:44.948+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:45.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:44.948+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:45.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:44.948+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:45.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:44.949+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:45.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:45.252+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:45.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:45.252+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:45.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:45.252+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:45.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:45.253+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:46.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:38:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:38:45.742+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21209 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:38:46.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:45.844+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:46.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:45.844+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:46.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:45.844+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:46.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:45.845+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:46.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:45.973+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:46.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:45.973+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:46.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:45.973+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:46.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:45.974+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:46.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:46.247+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:46.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:46.247+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:46.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:46.247+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:46.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:46.248+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:46.954 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:46.893+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:46.955 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:46.893+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:46.955 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:46.893+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:46.955 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:46.893+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:47.213 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:46.954+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:47.213 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:46.954+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:47.213 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:46.954+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:47.213 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:46.955+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:47.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:47.212+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:47.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:47.212+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:47.212+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:47.213+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:48.212 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:47.996+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:48.212 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:47.996+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:48.213 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:47.996+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:48.213 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:47.997+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:48.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:47.873+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:48.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:47.873+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:48.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:47.873+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:48.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:47.874+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:48.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:48.211+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:48.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:48.211+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:48.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:48.211+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:48.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:48.212+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:49.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:49.024+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:49.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:49.024+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:49.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:49.024+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:49.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:49.025+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:49.171 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:48.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:49.171 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:48.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:49.171 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:48.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:49.171 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:48.867+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:49.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:49.169+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:49.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:49.169+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:49.169+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:49.170+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:50.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:50.009+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:50.009+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:50.009+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:50.010+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:49.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:49.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:49.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:49.828+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:50.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:50.188+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:50.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:50.188+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:50.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:50.188+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:50.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:50.188+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:51.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:38:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:38:50.744+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21299 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:38:51.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:50.850+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:51.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:50.850+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:51.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:50.850+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:51.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:50.851+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:51.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:50.975+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:51.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:50.975+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:51.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:50.975+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:51.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:50.976+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:51.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:51.211+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:51.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:51.211+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:51.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:51.211+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:51.212+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:52.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:51.873+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:52.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:51.873+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:51.873+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:51.874+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:51.957+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:51.957+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:51.957+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:51.958+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:52.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:52.173+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:52.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:52.173+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:52.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:52.173+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:52.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:52.174+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:53.212 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:52.946+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:53.213 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:52.946+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:53.213 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:52.946+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:53.213 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:52.947+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:53.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:52.856+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:53.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:52.856+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:53.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:52.856+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:53.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:52.857+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:53.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:53.211+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:53.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:53.211+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:53.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:53.211+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:53.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:53.212+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:54.261 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:53.953+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:54.261 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:53.953+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:54.261 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:53.953+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:54.261 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:53.954+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:54.261 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:53.896+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:54.261 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:53.896+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:54.262 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:53.896+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:54.262 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:53.897+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:54.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:54.260+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:54.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:54.260+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:54.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:54.260+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:54.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:54.261+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:55.273 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:54.989+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:55.274 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:54.989+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:55.274 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:54.989+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:55.274 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:54.990+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:55.274 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:54.879+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:55.274 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:54.879+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:55.275 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:54.879+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:55.275 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:54.880+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:55.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:55.269+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:55.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:55.269+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:55.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:55.269+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:55.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:55.270+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:56.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:38:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:38:55.747+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21391 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:38:56.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:55.984+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:56.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:55.984+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:56.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:55.984+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:56.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:55.985+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:56.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:55.923+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:56.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:55.923+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:56.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:55.923+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:56.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:55.924+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:56.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:56.271+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:56.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:56.271+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:56.271+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:56.271+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:57.230 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:56.949+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:57.231 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:56.949+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:57.231 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:56.949+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:57.231 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:56.950+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:57.231 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:56.911+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:57.231 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:56.911+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:57.231 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:56.911+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:57.231 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:56.912+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:57.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:57.229+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:57.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:57.229+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:57.229+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:57.230+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:58.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:57.945+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:58.273 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:57.945+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:58.273 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:57.945+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:58.273 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:57.945+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:58.273 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:57.962+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:58.273 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:57.962+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:58.273 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:57.962+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:58.273 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:57.963+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:58.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:58.271+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:58.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:58.271+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:58.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:58.271+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:58.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:58.272+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:59.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:58.941+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:59.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:58.941+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:59.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:58.941+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:38:59.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:58.942+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:38:59.247 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:58.918+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:59.247 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:58.918+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:59.247 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:58.918+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:38:59.247 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:58.919+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:38:59.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:59.321+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:59.321+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:59.321+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:38:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:38:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:38:59.322+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:00.181 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:59.926+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:00.181 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:59.926+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:00.181 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:59.926+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:00.181 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:38:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:38:59.927+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:00.181 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:59.900+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:00.181 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:59.900+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:00.182 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:59.900+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:00.182 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:38:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:38:59.900+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:00.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:00.342+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:00.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:00.342+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:00.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:00.342+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:00.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:00.343+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:01.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:39:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:39:00.749+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21480 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:39:01.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:00.924+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:01.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:00.924+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:01.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:00.924+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:01.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:00.924+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:00.970+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:00.970+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:00.970+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:00.971+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:01.540 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:01.375+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:01.540 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:01.375+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:01.540 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:01.375+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:01.540 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:01.376+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:02.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:01.948+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:01.948+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:01.948+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:01.948+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:02.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:01.926+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:02.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:01.926+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:02.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:01.926+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:02.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:01.927+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:02.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:02.413+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:02.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:02.413+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:02.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:02.413+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:02.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:02.414+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:03.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:02.948+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:02.948+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:02.948+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:02.949+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:02.913+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:02.913+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:02.913+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:02.914+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:03.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:03.419+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:03.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:03.419+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:03.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:03.419+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:03.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:03.420+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:04.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:03.944+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:04.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:03.944+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:04.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:03.944+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:04.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:03.945+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:04.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:03.893+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:04.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:03.893+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:04.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:03.893+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:04.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:03.894+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:04.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:04.435+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:04.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:04.435+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:04.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:04.435+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:04.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:04.436+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:05.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:04.920+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:04.920+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:04.920+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:04.921+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:05.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:04.937+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:05.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:04.937+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:05.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:04.937+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:05.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:04.937+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:05.752 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:05.459+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:05.752 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:05.459+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:05.752 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:05.459+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:05.753 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:05.460+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:06.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:39:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:39:05.751+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21572 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:39:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:05.932+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:05.932+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:05.932+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:05.933+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:05.970+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:05.970+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:05.970+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:05.971+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:06.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:06.432+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:06.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:06.432+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:06.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:06.432+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:06.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:06.433+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:07.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:06.962+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:07.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:06.962+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:07.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:06.962+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:07.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:06.962+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:07.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:06.989+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:07.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:06.989+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:07.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:06.989+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:07.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:06.990+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:07.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:07.460+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:07.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:07.460+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:07.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:07.460+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:07.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:07.461+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:08.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:07.980+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:08.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:07.980+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:08.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:07.980+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:08.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:07.980+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:08.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:08.025+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:08.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:08.025+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:08.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:08.025+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:08.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:08.025+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:08.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:08.490+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:08.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:08.490+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:08.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:08.490+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:08.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:08.491+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:09.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:09.005+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:09.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:09.005+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:09.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:09.005+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:09.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:09.006+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:09.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:09.052+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:09.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:09.052+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:09.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:09.052+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:09.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:09.053+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:09.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:09.502+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:09.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:09.502+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:09.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:09.502+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:09.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:09.502+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:10.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:09.981+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:10.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:09.981+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:10.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:09.981+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:10.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:09.982+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:10.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:10.067+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:10.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:10.067+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:10.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:10.067+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:10.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:10.068+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:10.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:39:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:39:10.753+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21662 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:39:10.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:10.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:10.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:10.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:10.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:10.530+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:10.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:10.531+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:11.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:11.006+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:11.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:11.006+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:11.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:11.006+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:11.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:11.007+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:11.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:11.093+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:11.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:11.093+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:11.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:11.093+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:11.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:11.094+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:11.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:11.566+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:11.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:11.566+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:11.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:11.566+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:11.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:11.566+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:12.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:11.968+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:12.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:11.968+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:12.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:11.968+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:12.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:11.969+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:12.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:12.105+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:12.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:12.105+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:12.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:12.105+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:12.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:12.106+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:12.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:12.577+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:12.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:12.577+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:12.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:12.577+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:12.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:12.578+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:13.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:13.007+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:13.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:13.007+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:13.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:13.007+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:13.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:13.008+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:13.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:13.123+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:13.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:13.123+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:13.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:13.123+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:13.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:13.124+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:13.962 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:13.617+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:13.962 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:13.617+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:13.962 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:13.617+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:13.962 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:13.618+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:14.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:13.961+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:14.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:13.961+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:13.961+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:13.962+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:14.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:14.154+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:14.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:14.154+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:14.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:14.154+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:14.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:14.155+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:14.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:14.569+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:14.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:14.569+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:14.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:14.569+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:14.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:14.569+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:15.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:14.997+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:15.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:14.997+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:15.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:14.997+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:15.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:14.998+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:15.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:15.176+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:15.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:15.176+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:15.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:15.176+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:15.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:15.177+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:15.883 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:39:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:39:15.756+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21753 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:39:15.883 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:15.590+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:15.883 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:15.590+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:15.883 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:15.590+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:15.883 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:15.591+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:16.166 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:16.022+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:16.167 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:16.022+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:16.167 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:16.022+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:16.167 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:16.022+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:16.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:16.166+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:16.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:16.166+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:16.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:16.166+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:16.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:16.166+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:16.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:16.617+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:16.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:16.617+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:16.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:16.617+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:16.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:16.618+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:17.155 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:17.016+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:17.155 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:17.017+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:17.155 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:17.017+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:17.155 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:17.018+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:17.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:17.154+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:17.154+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:17.154+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:17.154+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:17.924 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:17.661+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:17.925 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:17.661+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:17.925 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:17.661+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:17.925 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:17.662+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:18.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:18.203+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:18.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:18.203+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:18.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:18.203+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:18.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:18.204+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:18.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:18.037+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:18.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:18.037+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:18.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:18.037+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:18.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:18.037+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:19.074 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:18.686+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:19.074 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:18.686+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:19.074 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:18.686+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:19.075 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:18.687+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:19.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:19.166+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:19.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:19.166+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:19.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:19.166+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:19.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:19.166+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:19.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:19.073+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:19.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:19.073+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:19.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:19.073+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:19.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:19.074+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:19.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:19.699+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:19.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:19.699+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:19.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:19.699+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:19.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:19.700+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:20.086+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:20.086+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:20.086+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:20.087+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:20.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:20.169+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:20.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:20.169+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:20.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:20.169+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:20.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:20.170+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:21.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:39:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:39:20.758+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21842 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:39:21.069 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:20.673+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:21.069 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:20.673+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:21.069 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:20.673+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:21.069 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:20.673+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:21.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:21.067+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:21.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:21.067+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:21.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:21.067+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:21.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:21.068+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:21.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:21.210+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:21.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:21.210+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:21.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:21.210+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:21.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:21.211+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:22.067 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:21.632+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:22.068 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:21.632+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:22.068 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:21.632+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:22.068 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:21.633+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:22.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:22.203+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:22.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:22.203+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:22.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:22.203+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:22.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:22.204+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:22.066+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:22.066+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:22.066+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:22.067+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:23.044 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:22.658+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:23.044 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:22.658+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:23.044 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:22.658+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:23.044 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:22.659+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:23.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:23.051+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:23.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:23.051+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:23.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:23.051+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:23.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:23.052+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:23.213+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:23.213+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:23.213+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:23.214+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:24.068 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:23.634+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:24.068 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:23.634+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:24.068 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:23.634+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:24.068 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:23.635+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:24.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:24.067+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:24.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:24.067+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:24.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:24.067+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:24.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:24.068+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:24.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:24.248+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:24.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:24.248+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:24.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:24.248+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:24.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:24.249+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:24.665+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:24.665+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:24.665+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:24.666+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:25.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:25.117+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:25.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:25.117+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:25.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:25.117+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:25.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:25.117+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:25.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:25.277+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:25.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:25.277+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:25.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:25.277+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:25.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:25.279+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:25.884 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:39:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:39:25.760+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21933 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:39:25.884 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:25.633+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:25.884 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:25.633+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:25.884 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:25.633+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:25.884 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:25.633+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:26.290 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:26.118+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:26.290 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:26.118+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:26.290 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:26.118+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:26.290 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:26.119+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:26.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:26.289+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:26.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:26.289+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:26.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:26.289+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:26.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:26.290+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:27.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:26.649+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:27.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:26.649+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:27.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:26.649+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:27.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:26.650+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:27.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:27.246+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:27.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:27.246+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:27.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:27.246+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:27.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:27.247+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:27.076+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:27.076+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:27.076+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:27.077+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:28.084 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:27.636+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:28.084 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:27.636+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:28.084 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:27.636+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:28.084 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:27.637+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:28.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:28.215+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:28.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:28.215+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:28.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:28.215+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:28.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:28.216+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:28.083+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:28.083+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:28.083+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:28.084+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:29.067 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:28.619+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:29.068 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:28.619+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:29.068 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:28.619+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:29.068 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:28.619+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:29.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:29.253+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:29.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:29.253+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:29.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:29.253+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:29.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:29.253+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:29.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:29.066+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:29.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:29.066+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:29.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:29.066+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:29.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:29.067+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:29.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:29.632+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:29.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:29.632+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:29.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:29.632+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:29.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:29.632+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:30.182 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:30.036+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:30.182 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:30.036+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:30.182 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:30.036+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:30.182 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:30.037+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:30.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:30.290+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:30.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:30.290+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:30.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:30.290+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:30.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:30.291+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:31.041 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:39:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:39:30.763+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22022 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:39:31.041 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:30.662+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:31.042 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:30.662+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:31.042 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:30.662+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:31.042 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:30.663+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:31.293 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:31.040+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:31.293 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:31.040+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:31.293 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:31.040+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:31.293 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:31.041+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:31.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:31.292+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:31.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:31.292+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:31.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:31.292+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:31.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:31.293+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:32.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:31.707+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:32.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:31.707+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:32.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:31.707+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:32.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:31.707+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:32.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:32.331+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:32.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:32.331+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:32.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:32.331+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:32.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:32.331+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:32.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:32.086+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:32.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:32.086+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:32.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:32.086+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:32.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:32.087+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:33.041 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:32.663+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:33.041 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:32.663+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:33.041 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:32.663+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:33.041 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:32.664+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:33.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:33.040+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:33.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:33.040+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:33.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:33.040+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:33.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:33.040+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:33.629 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:33.357+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:33.629 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:33.357+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:33.629 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:33.357+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:33.629 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:33.357+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:34.083 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:33.628+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:34.083 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:33.628+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:34.083 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:33.628+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:34.084 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:33.628+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:34.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:34.082+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:34.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:34.082+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:34.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:34.082+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:34.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:34.083+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:34.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:34.595+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:34.595+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:34.595+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:34.595+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:34.349+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:34.349+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:34.349+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:34.350+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:35.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:35.101+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:35.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:35.101+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:35.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:35.101+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:35.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:35.102+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:35.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:35.348+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:35.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:35.348+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:35.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:35.348+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:35.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:35.349+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:35.889 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:39:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:39:35.765+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22113 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:39:35.889 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:35.606+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:35.889 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:35.606+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:35.889 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:35.606+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:35.889 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:35.606+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:36.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:36.083+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:36.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:36.083+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:36.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:36.083+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:36.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:36.084+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:36.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:36.374+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:36.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:36.374+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:36.374+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:36.374+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:36.609+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:36.609+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:36.609+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:36.610+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:37.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:37.051+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:37.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:37.051+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:37.051+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:37.052+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:37.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:37.601+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:37.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:37.601+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:37.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:37.601+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:37.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:37.602+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:37.423+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:37.423+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:37.423+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:37.424+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:38.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:38.022+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:38.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:38.022+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:38.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:38.022+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:38.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:38.023+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:38.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:38.642+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:38.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:38.642+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:38.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:38.642+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:38.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:38.643+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:38.443+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:38.443+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:38.443+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:38.444+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:39.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:38.990+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:39.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:38.990+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:39.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:38.990+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:39.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:38.991+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:39.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:39.414+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:39.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:39.414+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:39.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:39.414+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:39.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:39.415+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:40.019 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:39.606+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:40.019 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:39.606+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:40.019 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:39.606+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:40.019 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:39.607+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:40.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:40.019+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:40.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:40.019+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:40.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:40.019+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:40.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:40.019+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:40.769 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:40.451+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:40.769 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:40.451+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:40.769 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:40.451+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:40.769 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:40.451+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:40.769 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:40.588+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:40.769 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:40.588+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:40.769 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:40.588+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:40.769 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:40.588+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:41.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:39:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:39:40.767+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22200 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:39:41.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:41.000+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:41.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:41.000+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:41.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:41.000+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:41.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:41.001+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:41.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:41.501+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:41.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:41.501+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:41.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:41.501+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:41.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:41.502+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:41.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:41.576+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:41.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:41.576+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:41.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:41.576+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:41.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:41.577+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:42.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:41.973+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:42.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:41.973+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:42.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:41.973+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:42.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:41.974+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:42.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:42.475+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:42.475+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:42.475+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:42.476+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:42.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:42.607+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:42.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:42.607+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:42.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:42.607+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:42.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:42.607+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:42.995+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:42.995+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:42.995+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:42.996+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:43.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:43.465+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:43.465+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:43.465+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:43.466+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:43.656+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:43.656+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:43.656+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:43.657+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:44.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:43.959+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:43.959+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:43.959+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:44.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:43.960+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:44.450+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:44.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:44.450+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:44.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:44.450+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:44.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:44.451+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:44.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:44.662+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:44.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:44.662+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:44.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:44.662+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:44.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:44.662+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:45.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:44.927+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:45.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:44.928+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:45.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:44.928+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:45.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:44.928+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:45.770 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:45.438+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:45.770 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:45.438+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:45.770 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:45.438+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:45.773 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:45.439+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:45.773 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:45.670+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:45.773 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:45.670+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:45.773 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:45.670+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:45.773 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:45.671+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:46.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:39:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:39:45.770+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22292 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:39:46.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:45.931+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:46.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:45.931+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:46.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:45.931+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:46.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:45.932+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:46.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:46.460+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:46.460+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:46.460+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:46.461+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:46.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:46.695+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:46.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:46.695+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:46.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:46.695+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:46.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:46.696+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:47.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:46.885+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:47.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:46.885+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:47.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:46.885+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:47.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:46.885+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:47.744 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:47.494+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:47.744 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:47.494+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:47.744 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:47.494+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:47.745 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:47.495+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:47.745 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:47.743+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:47.745 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:47.743+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:47.745 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:47.743+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:48.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:47.744+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:48.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:47.851+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:48.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:47.851+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:48.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:47.851+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:48.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:47.852+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:48.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:48.501+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:48.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:48.501+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:48.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:48.501+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:48.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:48.502+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:48.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:48.718+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:48.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:48.718+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:48.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:48.718+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:48.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:48.719+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:49.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:48.861+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:49.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:48.861+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:49.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:48.861+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:49.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:48.862+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:49.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:49.465+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:49.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:49.465+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:49.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:49.465+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:49.466+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:50.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:49.890+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:49.890+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:49.890+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:49.890+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:49.702+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:49.702+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:49.702+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:49.702+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:50.774 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:50.513+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:50.774 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:50.513+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:50.774 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:50.513+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:50.774 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:50.514+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:50.774 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:50.710+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:50.774 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:50.710+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:50.774 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:50.710+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:50.774 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:50.711+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:51.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:39:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:39:50.772+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22382 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:39:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:50.843+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:50.843+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:50.843+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:50.843+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:51.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:51.675+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:51.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:51.675+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:51.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:51.675+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:51.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:51.675+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:51.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:51.525+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:51.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:51.525+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:51.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:51.525+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:51.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:51.526+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:52.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:51.844+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:52.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:51.844+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:52.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:51.844+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:52.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:51.845+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:52.826 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:52.643+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:52.826 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:52.643+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:52.826 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:52.643+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:52.826 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:52.644+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:52.826 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:52.485+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:52.827 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:52.485+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:52.827 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:52.485+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:52.827 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:52.486+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:53.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:52.825+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:53.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:52.825+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:53.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:52.825+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:53.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:52.826+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:53.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:53.498+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:53.498+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:53.498+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:53.499+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:53.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:53.687+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:53.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:53.687+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:53.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:53.687+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:53.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:53.688+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:54.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:53.855+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:54.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:53.855+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:53.855+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:53.856+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:54.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:54.536+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:54.536+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:54.536+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:54.538+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:54.728+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:54.728+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:54.728+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:54.728+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:55.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:54.837+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:55.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:54.837+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:55.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:54.837+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:55.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:54.838+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:55.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:39:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:39:55.775+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22474 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:39:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:55.731+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:55.731+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:55.731+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:55.732+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:55.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:55.811+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:55.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:55.811+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:55.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:55.811+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:55.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:55.812+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:55.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:55.571+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:55.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:55.571+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:55.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:55.571+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:55.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:55.572+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:56.764+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:57.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:56.764+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:57.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:56.764+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:57.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:56.765+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:57.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:56.811+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:57.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:56.811+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:57.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:56.811+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:57.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:56.811+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:57.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:56.614+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:57.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:56.614+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:57.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:56.614+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:57.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:56.614+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:57.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:57.774+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:57.774+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:57.774+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:57.775+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:57.761+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:57.761+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:57.761+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:57.762+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:57.585+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:57.585+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:57.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:57.585+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:57.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:57.585+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:39:59.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:58.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:58.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:58.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:39:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:58.756+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:59.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:58.804+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:59.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:58.804+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:59.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:58.804+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:39:59.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:58.805+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:39:59.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:58.618+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:59.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:58.618+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:59.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:58.618+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:39:59.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:58.618+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:00.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:59.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:00.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:59.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:00.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:59.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:00.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:39:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:39:59.604+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:59.795+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:59.795+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:59.795+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:39:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:39:59.795+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:59.779+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:59.779+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:00.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:59.779+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:00.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:39:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:39:59.780+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:00.904 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:40:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:40:00.777+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22562 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:40:00.904 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:00.800+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:00.904 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:00.800+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:00.904 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:00.800+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:00.904 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:00.800+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:00.905 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:00.640+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:00.905 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:00.640+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:00.905 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:00.640+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:00.905 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:00.640+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:00.905 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:00.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:00.905 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:00.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:00.905 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:00.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:00.905 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:00.818+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:01.864 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:01.756+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:01.864 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:01.756+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:01.864 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:01.756+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:01.865 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:01.757+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:01.865 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:01.599+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:01.865 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:01.599+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:01.865 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:01.599+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:01.865 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:01.600+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:02.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:01.863+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:02.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:01.863+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:01.863+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:01.864+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:02.876 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:02.796+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:02.876 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:02.796+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:02.876 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:02.796+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:02.876 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:02.797+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:02.877 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:02.590+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:02.877 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:02.590+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:02.877 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:02.590+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:02.877 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:02.591+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:02.875+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:02.875+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:02.875+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:02.876+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:03.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:03.796+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:03.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:03.796+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:03.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:03.796+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:03.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:03.796+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:03.580+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:03.580+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:03.580+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:03.580+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:04.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:03.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:04.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:03.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:04.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:03.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:04.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:03.849+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:04.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:04.575+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:04.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:04.575+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:04.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:04.575+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:04.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:04.576+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:05.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:04.836+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:05.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:04.836+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:05.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:04.836+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:05.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:04.837+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:05.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:04.874+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:05.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:04.874+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:05.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:04.874+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:05.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:04.874+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:05.891 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:40:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:40:05.779+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22654 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:40:05.892 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:05.837+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:05.892 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:05.837+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:05.892 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:05.837+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:05.892 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:05.838+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:05.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:05.590+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:05.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:05.590+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:05.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:05.590+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:05.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:05.591+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:06.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:05.890+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:06.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:05.890+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:06.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:05.890+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:06.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:05.891+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:06.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:06.563+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:06.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:06.563+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:06.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:06.563+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:06.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:06.564+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:07.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:06.839+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:07.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:06.839+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:07.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:06.839+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:07.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:06.839+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:07.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:06.886+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:07.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:06.886+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:07.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:06.886+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:07.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:06.887+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:07.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:07.523+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:07.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:07.523+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:07.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:07.523+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:07.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:07.524+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:08.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:07.842+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:08.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:07.842+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:08.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:07.842+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:08.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:07.843+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:08.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:07.931+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:08.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:07.931+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:08.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:07.931+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:08.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:07.931+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:08.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:08.529+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:08.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:08.529+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:08.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:08.529+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:08.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:08.530+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:09.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:08.875+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:09.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:08.875+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:09.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:08.875+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:09.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:08.875+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:09.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:08.923+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:09.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:08.923+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:09.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:08.923+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:09.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:08.923+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:09.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:09.526+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:09.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:09.526+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:09.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:09.526+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:09.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:09.527+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:10.184 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:09.906+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:10.185 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:09.906+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:10.185 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:09.906+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:10.185 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:09.907+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:10.185 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:09.932+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:10.185 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:09.932+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:10.185 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:09.932+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:10.185 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:09.933+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:10.783 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:10.485+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:10.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:10.485+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:10.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:10.485+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:10.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:10.486+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:11.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:40:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:40:10.782+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22744 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:40:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:10.979+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:10.979+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:10.979+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:10.980+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:11.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:10.870+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:11.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:10.870+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:11.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:10.870+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:11.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:10.871+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:11.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:11.528+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:11.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:11.528+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:11.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:11.528+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:11.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:11.528+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:12.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:11.898+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:12.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:11.898+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:12.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:11.898+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:12.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:11.898+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:12.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:12.009+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:12.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:12.009+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:12.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:12.009+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:12.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:12.010+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:12.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:12.508+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:12.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:12.508+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:12.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:12.508+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:12.509+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:13.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:12.970+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:13.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:12.970+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:13.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:12.970+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:13.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:12.970+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:13.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:12.917+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:13.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:12.917+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:13.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:12.917+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:13.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:12.918+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:13.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:13.521+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:13.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:13.521+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:13.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:13.521+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:13.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:13.522+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:13.945+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:13.945+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:13.945+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:14.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:13.945+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:14.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:13.986+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:14.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:13.986+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:14.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:13.986+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:14.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:13.987+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:14.559+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:14.559+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:14.559+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:14.560+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:15.268 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:14.920+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:15.268 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:14.920+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:15.268 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:14.920+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:15.268 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:14.920+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:15.269 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:14.941+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:15.269 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:14.941+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:15.269 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:14.941+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:15.269 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:14.942+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:15.909 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:40:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:40:15.784+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22836 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:40:15.909 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:15.590+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:15.909 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:15.590+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:15.909 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:15.590+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:15.909 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:15.592+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:16.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:15.913+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:16.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:15.913+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:16.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:15.913+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:16.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:15.914+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:16.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:15.933+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:16.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:15.933+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:16.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:15.933+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:16.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:15.934+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:16.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:16.607+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:16.887 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:16.607+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:16.887 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:16.607+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:16.887 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:16.607+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:17.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:16.905+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:17.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:16.905+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:17.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:16.905+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:17.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:16.906+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:17.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:16.885+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:17.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:16.885+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:17.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:16.885+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:17.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:16.886+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:17.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:17.593+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:17.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:17.593+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:17.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:17.593+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:17.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:17.594+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:18.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:17.927+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:18.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:17.927+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:18.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:17.927+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:18.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:17.927+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:18.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:17.915+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:18.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:17.915+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:18.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:17.915+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:18.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:17.916+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:18.884 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:18.608+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:18.884 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:18.608+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:18.884 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:18.608+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:18.884 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:18.609+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:19.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:18.906+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:19.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:18.906+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:19.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:18.906+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:19.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:18.907+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:19.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:18.883+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:19.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:18.883+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:19.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:18.883+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:19.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:18.883+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:19.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:19.578+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:19.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:19.578+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:19.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:19.578+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:19.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:19.578+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:20.185 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:19.874+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:20.185 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:19.874+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:20.185 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:19.874+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:20.185 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:19.875+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:20.185 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:19.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:20.186 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:19.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:20.186 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:19.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:20.186 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:19.858+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:20.914 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:40:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:40:20.787+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22925 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:40:20.915 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:20.853+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:20.915 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:20.853+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:20.915 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:20.853+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:20.915 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:20.854+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:20.915 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:20.847+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:20.915 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:20.847+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:20.915 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:20.847+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:20.915 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:20.849+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:20.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:20.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:20.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:20.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:20.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:20.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:20.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:20.616+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:21.718 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:21.591+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:21.718 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:21.591+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:21.718 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:21.591+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:21.718 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:21.592+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:21.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:21.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:21.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:22.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:21.818+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:22.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:21.886+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:22.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:21.886+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:22.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:21.886+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:22.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:21.887+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:22.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:22.579+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:22.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:22.579+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:22.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:22.579+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:22.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:22.580+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:22.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:22.769+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:22.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:22.769+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:22.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:22.769+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:22.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:22.769+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:23.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:22.910+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:23.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:22.910+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:23.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:22.910+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:23.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:22.911+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:23.780 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:23.622+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:23.780 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:23.622+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:23.780 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:23.622+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:23.780 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:23.623+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:23.780 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:23.743+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:23.780 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:23.743+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:23.780 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:23.743+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:23.780 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:23.744+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:24.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:23.894+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:24.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:23.894+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:24.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:23.894+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:24.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:23.895+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:24.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:24.727+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:24.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:24.727+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:24.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:24.727+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:24.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:24.728+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:24.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:24.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:24.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:24.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:24.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:24.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:24.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:24.628+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:25.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:24.940+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:25.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:24.940+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:25.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:24.940+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:25.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:24.941+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:25.913 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:40:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:40:25.789+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23016 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:40:25.913 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:25.632+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:25.913 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:25.632+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:25.913 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:25.632+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:25.913 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:25.632+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:25.914 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:25.713+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:25.914 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:25.713+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:25.914 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:25.713+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:25.914 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:25.714+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:26.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:25.950+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:26.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:25.950+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:26.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:25.950+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:26.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:25.951+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:26.977 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:26.655+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:26.978 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:26.655+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:26.978 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:26.655+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:26.978 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:26.656+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:26.978 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:26.710+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:26.978 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:26.710+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:26.978 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:26.710+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:26.978 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:26.710+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:27.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:26.976+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:27.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:26.976+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:26.976+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:26.978+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:28.021 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:27.655+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:28.021 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:27.655+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:28.021 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:27.655+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:28.021 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:27.655+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:28.022 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:27.702+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:28.022 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:27.702+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:28.022 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:27.702+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:28.022 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:27.703+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:28.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:28.020+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:28.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:28.020+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:28.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:28.020+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:28.021+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:28.987 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:28.623+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:28.987 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:28.623+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:28.987 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:28.623+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:28.987 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:28.623+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:28.987 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:28.740+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:28.988 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:28.740+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:28.988 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:28.740+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:28.988 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:28.741+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:29.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:28.986+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:29.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:28.986+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:29.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:28.986+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:29.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:28.987+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:29.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:29.662+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:29.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:29.662+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:29.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:29.662+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:29.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:29.663+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:29.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:29.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:29.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:29.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:29.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:29.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:29.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:29.776+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:30.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:29.971+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:29.972+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:29.972+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:29.972+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:30.917 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:40:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:40:30.792+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23106 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:40:30.918 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:30.653+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:30.918 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:30.653+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:30.918 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:30.653+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:30.918 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:30.654+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:30.918 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:30.807+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:30.918 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:30.807+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:30.918 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:30.807+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:30.918 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:30.808+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:31.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:30.936+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:31.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:30.936+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:31.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:30.936+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:31.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:30.937+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:31.772+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:31.772+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:31.772+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:31.773+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:32.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:31.940+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:32.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:31.940+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:32.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:31.940+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:32.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:31.941+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:31.704+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:31.704+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:31.704+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:32.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:31.705+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:33.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:32.731+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:32.732+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:32.732+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:32.732+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:33.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:32.783+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:33.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:32.783+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:33.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:32.783+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:33.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:32.783+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:33.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:32.916+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:33.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:32.916+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:33.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:32.916+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:33.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:32.916+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:33.923 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:33.715+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:33.923 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:33.715+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:33.923 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:33.715+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:33.923 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:33.716+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:33.923 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:33.824+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:33.923 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:33.824+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:33.923 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:33.824+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:33.924 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:33.825+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:34.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:33.922+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:34.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:33.922+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:34.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:33.922+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:34.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:33.923+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:34.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:34.672+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:34.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:34.672+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:34.672+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:34.673+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:34.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:34.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:34.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:34.776+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:35.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:34.951+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:35.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:34.951+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:35.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:34.951+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:35.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:34.952+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:35.922 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:40:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:40:35.794+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23198 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:40:35.922 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:35.633+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:35.922 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:35.633+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:35.922 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:35.633+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:35.922 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:35.634+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:35.923 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:35.808+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:35.923 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:35.808+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:35.923 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:35.808+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:35.923 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:35.809+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:36.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:35.967+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:36.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:35.967+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:35.967+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:35.968+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:36.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:36.637+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:36.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:36.637+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:36.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:36.637+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:36.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:36.638+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:36.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:36.815+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:36.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:36.815+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:36.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:36.815+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:36.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:36.815+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:37.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:36.973+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:37.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:36.973+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:36.973+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:37.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:36.974+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:37.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:37.825+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:37.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:37.825+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:37.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:37.825+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:37.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:37.826+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:37.930 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:37.623+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:37.930 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:37.623+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:37.930 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:37.623+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:37.930 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:37.623+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:38.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:37.928+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:38.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:37.928+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:38.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:37.928+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:38.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:37.929+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:38.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:38.780+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:38.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:38.780+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:38.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:38.780+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:38.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:38.781+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:38.635+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:38.635+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:38.635+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:38.636+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:39.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:38.970+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:39.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:38.970+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:39.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:38.970+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:39.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:38.971+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:39.968 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:39.666+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:39.968 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:39.666+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:39.968 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:39.666+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:39.968 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:39.667+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:39.969 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:39.777+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:39.969 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:39.777+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:39.969 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:39.777+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:39.969 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:39.777+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:40.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:39.967+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:40.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:39.967+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:40.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:39.967+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:40.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:39.968+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:40.922 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:40:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:40:40.797+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23287 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:40:40.922 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:40.669+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:40.922 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:40.669+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:40.922 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:40.669+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:40.922 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:40.669+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:40.923 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:40.729+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:40.923 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:40.729+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:40.923 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:40.729+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:40.923 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:40.729+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:41.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:40.976+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:41.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:40.976+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:41.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:40.976+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:41.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:40.977+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:41.984 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:41.631+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:41.984 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:41.631+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:41.985 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:41.631+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:41.985 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:41.632+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:41.985 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:41.688+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:41.985 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:41.688+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:41.985 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:41.688+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:41.985 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:41.688+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:42.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:41.983+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:42.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:41.983+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:42.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:41.983+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:42.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:41.984+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:42.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:42.658+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:42.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:42.658+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:42.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:42.658+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:42.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:42.659+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:42.916 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:42.718+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:42.917 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:42.718+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:42.917 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:42.718+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:42.917 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:42.719+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:43.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:43.031+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:43.031+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:43.031+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:43.032+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:43.986 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:43.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:43.986 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:43.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:43.986 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:43.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:43.986 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:43.617+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:43.986 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:43.681+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:43.986 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:43.681+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:43.986 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:43.681+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:43.986 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:43.681+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:44.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:43.984+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:44.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:43.984+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:44.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:43.984+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:44.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:43.985+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:44.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:44.635+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:44.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:44.635+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:44.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:44.635+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:44.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:44.636+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:44.581+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:44.581+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:44.581+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:44.582+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:45.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:45.010+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:45.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:45.010+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:45.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:45.010+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:45.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:45.011+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:45.926 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:40:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:40:45.799+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23379 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:40:45.926 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:45.618+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:45.926 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:45.618+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:45.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:45.618+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:45.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:45.619+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:45.927 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:45.678+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:45.927 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:45.678+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:45.927 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:45.678+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:45.927 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:45.678+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:46.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:45.965+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:46.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:45.965+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:46.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:45.965+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:46.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:45.966+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:46.929 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:46.602+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:46.930 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:46.602+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:46.930 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:46.602+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:46.930 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:46.603+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:46.930 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:46.698+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:46.930 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:46.698+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:46.930 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:46.698+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:46.930 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:46.699+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:47.192 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:46.928+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:47.192 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:46.928+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:47.192 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:46.928+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:47.192 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:46.930+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:47.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:47.728+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:47.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:47.728+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:47.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:47.728+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:47.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:47.729+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:47.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:47.554+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:47.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:47.554+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:47.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:47.554+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:47.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:47.555+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:48.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:47.941+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:48.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:47.941+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:48.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:47.941+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:48.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:47.942+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:48.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:48.525+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:48.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:48.525+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:48.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:48.525+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:48.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:48.526+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:48.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:48.724+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:48.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:48.724+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:48.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:48.724+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:48.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:48.725+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:49.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:48.931+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:49.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:48.931+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:49.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:48.931+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:49.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:48.933+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:49.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:49.743+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:49.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:49.743+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:49.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:49.743+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:49.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:49.744+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:49.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:49.495+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:49.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:49.495+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:49.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:49.495+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:49.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:49.496+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:50.186 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:49.897+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:50.187 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:49.897+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:50.187 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:49.897+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:50.187 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:49.898+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:50.783 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:50.476+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:50.783 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:50.476+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:50.783 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:50.476+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:50.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:50.477+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:51.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:40:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:40:50.802+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23469 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:40:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:50.782+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:50.782+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:50.782+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:50.783+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:50.931+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:50.931+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:50.931+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:50.932+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-19T03:40:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:40:51.684+0000 7f9ccb836700 0 monclient(hunting): authenticate timed out after 300 2024-03-19T03:40:51.758 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:51.482+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:51.758 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:51.482+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:51.758 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:51.482+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:51.758 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:51.483+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:52.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:51.973+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:51.973+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:51.973+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:51.974+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:51.757+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:51.757+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:51.757+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:51.757+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:52.474 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -e sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 -- bash -c 'ceph orch ps' 2024-03-19T03:40:52.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:52.442+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:52.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:52.442+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:52.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:52.442+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:52.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:52.443+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:52.752 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/mon.smithi059/config 2024-03-19T03:40:52.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:52.720+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:52.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:52.720+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:52.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:52.720+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:52.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:52.721+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:53.279 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:52.972+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:53.279 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:52.972+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:53.279 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:52.972+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:53.279 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:52.973+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:53.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:53.477+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:53.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:53.477+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:53.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:53.477+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:53.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:53.478+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:53.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:53.675+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:53.954 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:53.675+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:53.954 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:53.675+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:53.954 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:53.675+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:54.269 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:53.952+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:54.269 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:53.952+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:54.269 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:53.952+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:54.269 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:53.953+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:54.525+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:54.525+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:54.525+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:54.526+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:54.689+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:54.689+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:54.689+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:54.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:54.690+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:55.204 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:54.953+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:55.205 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:54.953+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:55.205 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:54.953+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:55.205 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:54.953+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:55.806 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:55.478+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:55.806 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:55.478+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:55.806 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:55.478+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:55.806 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:55.479+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:55.807 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:55.683+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:55.807 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:55.683+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:55.807 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:55.683+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:55.807 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:55.684+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:56.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:40:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:40:55.804+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23560 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:40:56.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:56.001+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:56.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:56.001+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:56.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:56.001+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:56.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:56.002+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:56.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:56.447+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:56.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:56.447+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:56.447+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:56.448+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:56.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:56.679+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:56.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:56.679+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:56.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:56.679+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:56.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:56.680+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:57.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:56.954+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:57.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:56.954+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:57.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:56.954+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:57.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:56.955+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:57.475+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:57.475+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:57.475+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:57.476+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:57.634+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:57.634+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:57.634+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:57.635+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:58.263 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:57.956+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:58.263 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:57.956+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:58.263 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:57.956+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:58.263 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:57.956+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:58.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:58.449+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:58.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:58.449+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:58.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:58.449+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:58.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:58.450+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:58.999 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:58.669+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:58.999 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:58.669+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:58.999 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:58.669+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:59.000 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:58.670+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:59.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:58.999+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:59.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:58.999+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:59.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:58.999+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:40:59.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:40:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:40:58.999+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:40:59.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:59.496+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:59.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:59.496+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:59.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:59.496+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:40:59.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:40:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:40:59.496+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:40:59.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:59.685+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:59.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:59.685+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:59.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:59.685+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:40:59.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:40:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:40:59.685+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:00.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:00.000+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:00.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:00.000+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:00.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:00.000+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:00.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:00.001+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:00.808 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:00.545+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:00.808 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:00.545+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:00.808 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:00.545+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:00.808 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:00.546+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:00.809 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:00.643+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:00.809 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:00.643+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:00.809 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:00.643+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:00.809 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:00.643+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:01.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:41:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:41:00.807+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23648 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:41:01.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:01.032+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:01.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:01.032+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:01.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:01.032+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:01.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:01.032+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:01.667 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:01.512+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:01.667 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:01.512+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:01.667 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:01.512+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:01.667 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:01.513+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:02.015 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:01.667+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:02.015 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:01.667+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:02.015 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:01.667+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:02.015 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:01.668+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:02.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:02.015+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:02.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:02.015+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:02.015+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:02.016+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:02.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:02.640+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:02.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:02.640+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:02.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:02.640+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:02.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:02.641+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:02.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:02.513+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:02.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:02.513+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:02.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:02.513+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:02.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:02.514+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:03.282 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:03.047+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:03.282 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:03.047+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:03.282 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:03.047+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:03.282 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:03.047+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:03.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:03.560+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:03.560+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:03.560+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:03.561+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:03.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:03.678+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:03.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:03.678+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:03.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:03.678+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:03.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:03.679+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:04.257 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:03.998+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:04.257 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:03.998+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:04.257 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:03.998+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:04.258 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:03.999+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:04.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:04.564+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:04.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:04.564+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:04.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:04.564+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:04.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:04.565+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:04.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:04.633+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:04.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:04.633+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:04.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:04.633+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:04.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:04.633+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:05.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:04.996+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:05.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:04.996+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:04.996+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:05.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:04.997+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:05.811 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:05.523+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:05.811 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:05.523+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:05.811 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:05.523+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:05.811 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:05.524+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:05.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:05.623+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:05.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:05.623+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:05.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:05.623+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:05.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:05.623+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:06.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:41:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:41:05.810+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23739 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:41:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:05.967+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:05.967+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:05.967+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:05.967+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:06.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:06.579+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:06.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:06.579+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:06.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:06.579+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:06.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:06.580+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:06.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:06.485+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:06.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:06.485+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:06.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:06.485+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:06.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:06.486+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:07.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:06.982+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:07.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:06.982+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:07.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:06.982+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:07.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:06.983+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:07.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:07.507+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:07.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:07.507+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:07.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:07.507+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:07.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:07.508+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:07.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:07.605+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:07.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:07.605+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:07.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:07.605+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:07.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:07.606+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:08.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:08.019+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:08.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:08.019+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:08.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:08.019+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:08.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:08.020+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:08.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:08.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:08.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:08.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:08.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:08.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:08.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:08.493+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:08.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:08.623+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:08.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:08.623+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:08.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:08.623+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:08.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:08.624+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:09.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:09.060+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:09.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:09.060+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:09.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:09.060+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:09.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:09.061+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:09.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:09.500+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:09.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:09.500+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:09.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:09.500+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:09.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:09.501+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:09.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:09.648+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:09.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:09.648+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:09.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:09.648+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:09.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:09.649+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:10.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:10.061+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:10.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:10.061+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:10.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:10.061+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:10.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:10.062+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:10.813 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:10.681+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:10.813 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:10.681+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:10.813 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:10.681+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:10.813 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:10.681+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:10.814 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:10.464+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:10.814 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:10.464+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:10.814 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:10.464+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:10.814 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:10.465+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:11.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:41:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:41:10.812+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23830 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:41:11.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:11.061+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:11.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:11.061+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:11.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:11.061+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:11.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:11.062+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:11.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:11.431+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:11.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:11.431+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:11.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:11.431+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:11.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:11.432+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:12.021 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:11.731+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:12.022 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:11.731+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:12.022 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:11.731+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:12.022 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:11.731+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:12.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:12.020+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:12.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:12.020+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:12.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:12.020+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:12.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:12.021+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:12.729 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:12.394+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:12.729 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:12.394+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:12.729 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:12.394+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:12.729 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:12.395+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:12.728+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:12.728+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:13.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:12.728+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:13.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:12.729+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:12.975+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:12.975+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:12.975+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:12.976+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:13.637 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:13.384+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:13.637 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:13.384+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:13.637 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:13.384+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:13.637 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:13.385+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:14.002 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:13.707+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:14.002 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:13.707+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:14.002 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:13.707+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:14.002 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:13.707+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:14.319 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:14.000+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:14.319 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:14.000+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:14.320 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:14.000+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:14.320 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:14.001+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:14.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:14.371+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:14.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:14.371+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:14.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:14.371+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:14.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:14.371+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:15.046 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:14.710+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:15.046 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:14.710+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:15.046 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:14.710+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:15.046 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:14.711+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:15.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:15.045+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:15.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:15.045+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:15.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:15.045+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:15.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:15.045+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:15.705 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:15.409+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:15.705 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:15.409+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:15.705 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:15.409+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:15.705 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:15.409+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:16.033 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:41:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:41:15.815+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23936 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:41:16.033 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:15.704+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:16.033 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:15.704+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:16.033 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:15.704+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:16.033 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:15.705+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:16.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:16.032+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:16.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:16.032+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:16.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:16.032+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:16.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:16.033+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:16.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:16.689+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:16.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:16.689+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:16.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:16.689+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:16.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:16.689+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:16.452+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:16.452+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:16.452+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:16.453+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:17.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:17.012+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:17.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:17.012+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:17.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:17.012+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:17.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:17.013+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:17.708 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:17.435+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:17.708 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:17.435+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:17.708 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:17.435+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:17.708 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:17.436+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:18.030 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:17.707+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:18.030 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:17.707+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:18.030 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:17.707+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:18.030 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:17.708+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:18.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:18.029+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:18.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:18.029+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:18.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:18.029+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:18.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:18.029+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:18.742 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:18.394+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:18.742 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:18.394+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:18.743 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:18.394+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:18.743 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:18.395+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:19.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:18.741+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:19.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:18.741+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:19.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:18.741+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:19.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:18.742+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:19.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:19.076+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:19.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:19.076+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:19.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:19.076+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:19.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:19.077+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:19.780 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:19.404+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:19.781 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:19.404+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:19.781 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:19.404+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:19.781 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:19.405+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:20.032 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:20.030+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:20.032 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:20.030+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:20.032 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:20.030+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:20.032 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:20.031+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:20.032 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:19.790+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:20.032 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:19.790+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:20.032 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:19.790+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:20.032 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:19.791+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:20.765 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:20.436+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:20.765 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:20.436+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:20.765 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:20.436+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:20.765 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:20.437+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:20.765 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:20.764+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:20.766 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:20.764+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:20.766 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:20.764+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:21.033 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:41:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:41:20.817+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24026 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:41:21.033 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:20.765+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:21.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:21.032+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:21.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:21.032+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:21.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:21.032+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:21.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:21.033+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:21.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:21.393+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:21.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:21.393+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:21.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:21.393+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:21.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:21.393+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:22.071 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:21.795+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:22.072 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:21.795+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:22.072 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:21.795+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:22.072 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:21.796+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:22.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:22.071+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:22.071+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:22.071+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:22.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:22.071+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:22.786 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:22.434+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:22.786 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:22.434+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:22.786 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:22.434+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:22.787 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:22.435+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:22.785+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:23.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:22.785+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:23.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:22.785+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:23.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:22.786+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:23.479 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:23.116+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:23.480 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:23.116+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:23.480 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:23.116+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:23.480 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:23.117+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:23.765 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:23.478+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:23.765 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:23.478+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:23.765 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:23.478+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:23.765 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:23.479+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:24.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:23.764+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:24.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:23.764+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:24.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:23.764+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:23.764+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:24.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:24.162+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:24.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:24.162+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:24.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:24.162+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:24.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:24.163+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:24.805 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:24.444+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:24.805 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:24.444+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:24.805 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:24.444+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:24.805 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:24.445+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:24.804+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:24.804+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:24.804+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:24.805+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:25.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:25.442+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:25.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:25.442+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:25.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:25.442+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:25.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:25.443+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:25.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:25.199+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:25.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:25.199+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:25.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:25.199+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:25.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:25.200+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:26.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:41:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:41:25.820+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24117 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:41:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:25.767+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:25.767+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:25.767+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:25.767+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:26.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:26.474+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:26.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:26.474+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:26.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:26.474+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:26.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:26.474+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:26.239+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:26.239+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:26.239+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:26.240+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:27.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:26.771+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:27.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:26.771+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:27.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:26.771+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:27.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:26.772+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:27.496 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:27.211+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:27.496 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:27.211+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:27.496 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:27.211+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:27.496 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:27.212+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:27.820 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:27.495+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:27.820 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:27.495+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:27.821 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:27.495+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:27.821 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:27.496+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:28.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:27.819+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:28.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:27.819+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:28.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:27.819+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:28.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:27.820+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:28.541 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:28.251+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:28.541 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:28.251+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:28.541 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:28.251+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:28.541 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:28.251+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:28.799 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:28.540+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:28.799 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:28.540+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:28.799 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:28.540+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:28.799 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:28.541+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:28.798+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:28.798+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:28.798+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:28.799+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:29.563 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:29.273+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:29.563 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:29.273+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:29.563 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:29.273+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:29.563 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:29.274+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:29.832 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:29.562+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:29.832 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:29.562+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:29.832 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:29.562+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:29.832 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:29.563+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:29.831+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:29.831+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:30.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:29.831+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:30.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:29.831+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:30.578 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:30.236+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:30.579 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:30.236+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:30.579 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:30.236+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:30.579 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:30.237+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:30.836 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:41:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:41:30.823+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24208 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:41:30.836 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:30.578+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:30.836 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:30.578+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:30.836 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:30.578+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:30.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:30.578+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:30.837 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:30.800+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:30.837 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:30.800+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:30.837 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:30.800+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:30.837 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:30.801+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:31.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:31.210+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:31.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:31.210+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:31.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:31.210+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:31.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:31.211+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:31.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:31.586+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:31.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:31.586+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:31.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:31.586+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:31.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:31.587+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:31.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:31.830+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:31.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:31.830+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:31.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:31.830+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:31.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:31.831+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:32.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:32.239+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:32.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:32.239+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:32.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:32.239+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:32.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:32.240+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:32.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:32.545+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:32.545+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:32.545+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:32.546+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:33.206 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:32.873+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:33.206 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:32.873+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:33.206 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:32.873+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:33.206 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:32.873+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:33.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:33.205+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:33.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:33.205+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:33.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:33.205+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:33.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:33.205+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:33.891 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:33.595+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:33.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:33.595+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:33.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:33.595+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:33.893 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:33.595+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:34.159 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:33.890+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:34.159 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:33.890+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:34.159 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:33.890+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:34.159 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:33.891+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:34.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:34.158+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:34.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:34.158+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:34.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:34.158+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:34.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:34.159+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:34.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:34.620+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:34.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:34.620+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:34.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:34.620+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:34.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:34.621+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:35.140 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:34.885+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:35.140 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:34.885+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:35.141 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:34.885+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:35.141 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:34.885+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:35.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:35.191+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:35.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:35.191+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:35.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:35.191+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:35.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:35.192+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:35.849 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:41:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:41:35.825+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24299 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:41:35.849 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:35.595+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:35.849 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:35.595+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:35.850 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:35.595+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:35.850 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:35.596+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:36.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:35.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:36.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:35.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:36.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:35.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:36.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:35.857+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:36.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:36.199+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:36.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:36.199+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:36.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:36.199+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:36.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:36.200+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:36.904 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:36.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:36.904 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:36.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:36.904 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:36.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:36.904 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:36.607+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:37.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:36.903+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:37.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:36.903+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:37.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:36.903+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:37.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:36.904+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:37.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:37.200+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:37.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:37.200+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:37.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:37.200+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:37.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:37.201+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:37.867 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:37.596+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:37.867 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:37.596+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:37.867 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:37.596+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:37.867 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:37.597+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:38.194 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:37.866+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:38.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:37.866+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:38.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:37.866+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:38.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:37.867+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:38.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:38.193+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:38.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:38.193+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:38.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:38.193+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:38.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:38.194+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:39.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:38.642+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:38.642+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:38.642+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:38.643+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:38.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:38.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:38.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:38.858+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:39.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:39.240+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:39.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:39.240+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:39.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:39.240+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:39.241+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:40.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:39.834+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:40.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:39.834+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:39.834+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:39.834+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:39.613+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:39.613+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:39.613+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:39.614+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:40.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:40.267+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:40.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:40.267+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:40.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:40.267+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:40.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:40.268+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:40.953 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:41:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:41:40.829+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24389 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:41:40.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:40.826+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:40.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:40.826+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:40.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:40.826+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:40.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:40.827+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:40.953 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:40.655+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:40.954 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:40.655+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:40.954 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:40.655+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:40.954 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:40.656+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:41.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:41.286+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:41.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:41.286+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:41.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:41.286+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:41.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:41.287+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:42.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:41.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:42.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:41.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:42.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:41.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:42.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:41.858+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:42.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:41.683+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:42.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:41.683+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:42.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:41.683+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:42.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:41.684+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:42.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:42.319+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:42.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:42.319+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:42.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:42.319+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:42.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:42.320+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:43.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:42.678+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:43.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:42.678+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:43.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:42.678+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:43.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:42.678+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:43.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:42.902+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:43.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:42.902+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:43.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:42.902+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:43.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:42.903+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:43.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:43.331+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:43.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:43.331+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:43.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:43.331+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:43.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:43.332+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:43.867 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:43.669+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:43.867 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:43.669+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:43.868 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:43.669+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:43.868 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:43.670+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:43.866+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:43.866+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:43.866+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:43.867+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:44.709 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:44.368+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:44.709 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:44.368+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:44.710 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:44.368+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:44.710 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:44.369+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:45.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:44.708+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:45.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:44.708+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:45.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:44.708+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:45.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:44.709+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:44.891+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:44.891+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:44.891+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:44.892+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:45.737 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:45.380+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:45.737 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:45.380+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:45.737 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:45.380+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:45.738 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:45.381+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:46.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:41:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:41:45.831+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24480 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:41:46.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:45.931+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:46.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:45.931+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:46.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:45.931+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:46.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:45.932+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:46.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:45.736+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:46.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:45.736+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:46.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:45.736+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:46.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:45.737+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:46.755 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:46.374+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:46.755 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:46.374+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:46.755 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:46.374+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:46.755 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:46.375+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:47.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:46.754+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:47.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:46.754+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:47.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:46.754+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:47.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:46.755+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:46.969+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:46.969+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:46.969+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:46.970+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:47.730 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:47.360+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:47.730 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:47.360+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:47.730 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:47.360+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:47.730 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:47.361+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:48.015 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:47.729+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:48.015 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:47.729+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:48.015 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:47.729+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:48.015 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:47.730+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:48.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:48.014+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:48.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:48.014+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:48.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:48.014+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:48.015+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:48.781 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:48.363+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:48.781 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:48.363+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:48.781 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:48.363+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:48.781 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:48.364+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:48.780+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:49.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:48.780+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:49.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:48.780+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:49.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:48.780+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:49.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:49.004+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:49.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:49.004+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:49.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:49.004+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:49.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:49.005+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:49.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:49.355+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:49.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:49.355+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:49.355+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:49.356+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:49.997 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:49.741+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:49.997 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:49.741+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:49.997 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:49.741+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:49.998 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:49.742+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:50.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:49.996+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:50.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:49.996+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:50.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:49.996+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:50.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:49.997+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:50.744 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:50.402+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:50.744 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:50.402+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:50.745 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:50.402+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:50.745 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:50.403+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:50.999 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:41:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:41:50.834+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24571 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:41:50.999 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:50.743+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:50.999 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:50.743+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:50.999 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:50.743+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:50.999 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:50.745+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:50.998+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:50.998+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:50.998+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:50.999+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:51.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:51.379+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:51.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:51.379+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:51.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:51.379+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:51.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:51.379+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:51.998 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:51.745+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:51.998 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:51.745+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:51.998 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:51.745+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:51.998 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:51.746+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:52.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:51.997+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:52.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:51.997+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:52.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:51.997+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:52.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:51.998+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:52.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:52.332+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:52.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:52.332+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:52.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:52.332+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:52.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:52.333+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:52.962 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:52.709+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:52.962 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:52.709+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:52.962 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:52.709+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:52.962 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:52.709+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:53.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:52.961+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:53.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:52.961+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:53.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:52.961+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:53.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:52.961+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:53.685 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:53.379+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:53.685 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:53.379+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:53.686 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:53.379+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:53.686 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:53.381+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:54.085 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:53.684+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:54.085 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:53.684+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:54.085 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:53.684+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:54.085 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:53.685+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:54.085 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:53.913+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:54.085 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:53.913+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:54.085 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:53.913+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:54.086 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:53.914+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:54.667 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:54.372+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:54.667 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:54.372+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:54.667 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:54.372+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:54.667 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:54.373+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:54.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:54.666+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:54.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:54.666+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:54.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:54.666+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:54.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:54.667+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:55.328 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:54.949+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:55.328 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:54.950+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:55.328 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:54.950+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:55.328 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:54.950+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:55.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:55.327+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:55.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:55.327+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:55.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:55.327+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:55.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:55.327+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:55.964 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:41:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:41:55.837+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24662 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:41:55.964 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:55.927+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:55.965 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:55.927+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:55.965 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:55.927+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:55.965 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:55.928+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:55.965 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:55.713+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:55.965 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:55.713+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:55.965 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:55.713+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:55.965 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:55.714+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:56.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:56.312+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:56.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:56.312+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:56.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:56.312+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:56.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:56.313+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:56.897 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:56.667+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:56.897 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:56.667+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:56.897 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:56.667+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:56.897 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:56.668+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:57.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:56.895+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:57.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:56.895+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:56.895+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:56.896+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:57.719 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:57.360+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:57.720 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:57.360+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:57.720 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:57.360+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:57.720 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:57.361+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:58.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:57.718+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:58.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:57.718+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:58.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:57.718+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:58.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:57.719+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:57.874+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:57.874+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:57.874+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:57.875+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:58.743 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:58.370+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:58.743 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:58.370+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:58.743 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:58.370+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:58.744 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:58.371+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:59.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:58.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:59.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:58.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:59.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:58.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:41:59.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:58.743+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:41:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:58.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:58.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:58.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:41:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:58.853+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:41:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:59.365+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:59.365+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:59.365+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:41:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:41:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:41:59.365+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:00.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:59.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:00.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:59.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:00.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:59.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:00.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:41:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:41:59.792+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:59.842+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:59.842+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:59.842+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:41:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:41:59.843+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:00.375+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:00.375+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:00.375+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:00.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:00.376+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:01.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:42:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:42:00.839+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24751 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:42:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:00.805+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:00.805+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:00.805+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:00.806+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:00.850+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:00.850+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:00.850+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:00.851+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:01.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:01.365+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:01.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:01.365+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:01.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:01.365+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:01.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:01.365+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:01.875+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:01.875+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:02.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:01.875+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:02.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:01.876+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:01.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:01.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:01.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:01.828+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:02.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:02.330+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:02.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:02.330+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:02.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:02.330+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:02.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:02.331+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:03.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:02.863+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:02.863+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:02.863+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:02.864+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:02.845+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:02.845+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:02.845+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:02.846+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:03.810 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:03.363+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:03.810 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:03.363+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:03.810 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:03.363+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:03.810 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:03.364+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:04.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:03.809+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:04.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:03.809+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:04.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:03.809+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:04.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:03.809+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:04.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:03.855+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:04.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:03.855+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:04.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:03.855+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:04.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:03.856+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:04.760 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:04.355+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:04.760 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:04.355+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:04.760 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:04.355+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:04.761 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:04.356+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:04.816+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:04.816+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:04.816+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:04.817+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:04.759+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:04.759+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:04.759+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:04.760+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:05.765 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:05.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:05.765 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:05.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:05.765 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:05.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:05.765 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:05.361+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:06.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:42:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:42:05.842+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24841 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:42:06.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:05.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:06.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:05.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:06.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:05.764+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:06.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:05.764+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:05.796+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:05.796+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:05.796+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:05.797+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:06.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:06.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:06.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:06.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:06.796 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:06.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:06.796 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:06.359+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:07.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:06.808+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:06.808+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:06.808+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:06.809+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:07.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:06.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:07.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:06.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:07.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:06.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:07.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:06.795+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:07.329+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:07.329+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:07.329+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:07.330+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:08.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:07.841+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:08.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:07.841+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:07.841+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:07.842+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:07.754+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:07.754+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:07.754+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:07.755+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:08.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:08.293+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:08.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:08.293+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:08.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:08.293+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:08.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:08.294+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:09.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:08.890+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:08.890+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:08.890+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:08.891+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:08.762+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:08.762+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:08.762+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:08.762+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:09.266+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:09.266+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:09.266+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:09.267+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:10.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:09.850+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:10.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:09.850+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:10.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:09.850+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:10.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:09.851+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:09.769+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:09.769+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:09.769+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:09.770+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:10.230+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:10.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:10.230+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:10.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:10.230+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:10.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:10.231+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:11.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:42:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:42:10.845+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24932 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:42:11.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:10.881+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:11.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:10.881+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:11.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:10.881+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:11.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:10.881+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:10.733+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:10.733+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:10.733+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:10.734+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:11.229+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:11.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:11.229+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:11.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:11.229+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:11.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:11.230+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:12.004 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:11.914+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:12.004 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:11.914+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:12.004 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:11.914+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:12.004 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:11.915+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:12.005 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:11.716+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:12.005 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:11.716+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:12.005 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:11.716+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:12.005 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:11.716+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:12.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:12.262+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:12.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:12.262+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:12.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:12.262+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:12.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:12.263+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:12.751+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:12.751+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:12.751+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:13.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:12.752+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:12.867+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:12.867+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:12.867+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:12.867+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:13.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:13.304+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:13.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:13.304+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:13.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:13.304+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:13.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:13.305+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:13.899 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:13.753+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:13.899 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:13.753+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:13.899 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:13.753+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:13.899 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:13.754+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:14.285 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:13.898+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:14.285 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:13.898+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:14.285 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:13.898+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:14.285 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:13.899+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:14.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:14.284+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:14.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:14.284+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:14.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:14.284+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:14.285+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:15.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:14.947+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:14.947+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:14.947+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:14.948+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:14.720+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:14.720+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:14.720+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:14.720+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:15.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:15.235+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:15.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:15.235+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:15.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:15.235+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:15.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:15.237+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:15.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:42:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:42:15.847+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25023 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:42:15.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:15.901+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:15.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:15.901+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:15.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:15.901+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:15.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:15.902+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:15.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:15.682+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:15.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:15.682+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:15.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:15.682+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:15.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:15.683+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:16.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:16.240+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:16.240+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:16.240+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:16.241+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:16.667+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:17.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:16.667+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:17.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:16.667+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:17.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:16.667+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:16.916+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:16.916+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:16.916+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:16.917+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:17.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:17.277+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:17.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:17.277+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:17.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:17.277+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:17.278+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:17.942 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:17.691+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:17.942 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:17.691+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:17.942 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:17.691+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:17.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:17.691+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:17.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:17.940+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:17.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:17.940+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:17.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:17.940+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:17.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:17.941+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:18.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:18.268+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:18.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:18.268+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:18.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:18.268+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:18.269+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:18.987 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:18.642+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:18.987 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:18.642+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:18.987 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:18.642+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:18.987 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:18.643+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:19.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:18.986+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:19.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:18.986+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:19.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:18.986+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:19.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:18.987+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:19.251+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:19.251+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:19.251+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:19.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:19.251+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:20.011 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:19.597+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:20.011 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:19.597+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:20.011 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:19.597+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:20.011 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:19.597+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:20.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:20.238+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:20.238+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:20.238+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:20.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:20.240+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:20.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:20.010+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:20.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:20.010+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:20.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:20.010+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:20.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:20.011+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:20.975 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:42:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:42:20.850+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25113 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:42:20.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:20.629+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:20.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:20.629+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:20.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:20.629+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:20.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:20.629+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:21.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:21.207+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:21.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:21.207+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:21.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:21.207+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:21.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:21.208+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:21.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:20.995+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:21.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:20.995+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:21.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:20.995+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:21.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:20.995+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:21.955 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:21.596+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:21.955 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:21.596+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:21.955 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:21.596+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:21.955 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:21.597+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:22.233 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:21.955+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:22.233 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:21.955+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:22.233 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:21.955+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:22.234 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:21.956+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:22.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:22.233+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:22.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:22.233+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:22.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:22.233+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:22.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:22.234+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:22.916 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:22.648+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:22.917 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:22.648+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:22.917 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:22.648+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:22.917 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:22.649+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:23.267 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:22.917+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:23.267 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:22.917+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:23.267 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:22.917+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:23.267 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:22.918+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:23.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:23.267+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:23.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:23.267+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:23.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:23.267+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:23.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:23.268+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:23.937 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:23.653+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:23.938 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:23.653+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:23.938 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:23.653+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:23.938 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:23.653+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:24.286 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:23.937+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:24.286 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:23.937+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:24.286 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:23.937+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:24.286 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:23.938+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:24.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:24.286+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:24.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:24.286+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:24.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:24.286+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:24.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:24.287+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:24.926 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:24.639+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:24.926 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:24.639+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:24.926 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:24.639+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:24.927 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:24.640+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:25.244 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:24.926+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:25.244 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:24.926+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:25.244 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:24.926+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:25.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:24.927+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:25.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:25.244+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:25.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:25.244+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:25.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:25.244+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:25.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:25.245+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:25.863 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:42:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:42:25.854+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25204 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:42:25.864 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:25.612+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:25.864 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:25.612+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:25.864 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:25.612+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:25.864 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:25.613+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:26.231 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:25.945+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:26.231 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:25.945+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:26.231 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:25.945+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:26.231 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:25.945+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:26.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:26.231+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:26.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:26.231+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:26.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:26.231+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:26.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:26.232+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:26.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:26.640+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:26.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:26.640+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:26.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:26.640+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:26.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:26.640+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:27.251 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:26.949+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:27.251 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:26.949+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:27.251 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:26.949+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:27.251 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:26.950+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:27.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:27.251+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:27.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:27.251+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:27.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:27.251+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:27.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:27.252+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:27.938 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:27.598+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:27.938 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:27.598+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:27.938 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:27.598+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:27.938 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:27.598+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:28.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:27.938+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:28.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:27.938+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:28.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:27.938+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:28.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:27.939+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:28.574 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:28.213+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:28.574 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:28.213+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:28.575 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:28.213+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:28.575 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:28.214+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:28.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:28.574+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:28.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:28.575+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:28.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:28.575+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:28.575+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:29.216 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:28.946+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:29.216 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:28.946+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:29.216 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:28.946+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:29.216 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:28.947+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:29.555 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:29.216+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:29.555 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:29.216+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:29.555 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:29.216+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:29.555 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:29.217+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:29.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:29.554+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:29.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:29.554+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:29.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:29.554+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:29.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:29.555+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:30.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:30.206+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:30.206+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:30.206+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:30.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:30.207+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:30.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:29.964+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:30.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:29.964+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:30.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:29.964+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:30.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:29.965+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:30.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:30.598+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:30.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:30.598+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:30.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:30.598+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:30.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:30.599+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:31.255 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:42:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:42:30.855+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25295 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:42:31.255 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:30.965+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:31.255 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:30.965+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:31.255 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:30.965+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:31.255 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:30.966+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:31.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:31.254+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:31.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:31.254+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:31.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:31.254+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:31.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:31.255+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:31.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:31.611+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:31.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:31.611+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:31.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:31.611+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:31.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:31.612+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:32.262 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:31.974+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:32.262 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:31.974+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:32.263 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:31.974+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:32.263 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:31.974+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:32.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:32.261+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:32.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:32.261+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:32.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:32.261+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:32.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:32.263+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:32.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:32.572+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:32.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:32.572+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:32.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:32.572+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:32.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:32.573+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:33.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:33.010+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:33.273 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:33.010+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:33.273 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:33.010+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:33.273 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:33.011+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:33.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:33.271+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:33.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:33.271+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:33.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:33.271+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:33.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:33.272+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:33.962 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:33.609+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:33.962 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:33.609+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:33.962 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:33.609+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:33.962 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:33.610+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:34.281 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:33.961+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:34.281 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:33.961+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:34.281 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:33.961+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:34.282 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:33.962+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:34.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:34.280+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:34.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:34.280+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:34.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:34.280+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:34.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:34.281+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:35.002 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:34.625+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:35.002 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:34.625+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:35.002 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:34.625+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:35.002 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:34.626+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:35.263 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:35.000+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:35.263 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:35.000+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:35.264 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:35.000+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:35.264 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:35.002+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:35.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:35.262+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:35.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:35.262+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:35.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:35.262+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:35.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:35.263+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:35.884 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:42:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:42:35.858+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25386 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:42:35.884 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:35.628+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:35.884 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:35.628+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:35.884 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:35.628+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:35.885 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:35.628+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:36.252 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:35.999+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:36.252 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:35.999+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:36.252 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:35.999+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:36.252 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:36.000+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:36.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:36.251+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:36.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:36.251+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:36.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:36.251+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:36.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:36.252+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:36.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:36.605+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:36.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:36.605+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:36.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:36.605+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:36.605+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:37.295 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:36.955+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:37.295 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:36.955+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:37.296 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:36.955+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:37.296 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:36.956+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:37.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:37.294+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:37.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:37.294+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:37.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:37.294+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:37.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:37.295+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:37.908 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:37.628+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:37.908 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:37.628+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:37.908 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:37.628+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:37.908 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:37.628+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:38.327 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:37.907+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:38.327 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:37.907+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:38.327 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:37.907+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:38.327 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:37.908+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:38.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:38.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:38.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:38.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:38.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:38.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:38.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:38.327+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:38.954 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:38.624+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:38.954 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:38.624+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:38.954 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:38.624+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:38.954 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:38.625+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:39.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:38.952+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:39.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:38.952+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:39.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:38.952+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:39.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:38.954+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:39.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:39.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:39.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:39.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:39.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:39.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:39.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:39.326+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:39.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:39.647+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:39.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:39.647+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:39.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:39.647+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:39.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:39.648+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:40.286 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:39.946+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:40.286 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:39.946+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:40.286 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:39.946+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:40.286 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:39.947+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:40.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:40.285+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:40.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:40.285+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:40.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:40.285+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:40.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:40.286+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:40.928 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:42:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:42:40.861+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25476 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:42:40.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:40.668+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:40.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:40.668+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:40.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:40.668+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:40.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:40.669+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:41.265 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:40.927+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:41.266 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:40.927+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:41.266 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:40.927+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:41.266 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:40.928+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:41.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:41.264+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:41.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:41.264+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:41.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:41.264+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:41.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:41.265+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:41.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:41.646+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:41.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:41.646+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:41.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:41.646+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:41.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:41.646+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:42.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:41.950+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:42.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:41.950+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:42.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:41.950+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:42.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:41.950+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:42.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:42.271+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:42.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:42.271+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:42.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:42.271+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:42.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:42.272+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:42.990 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:42.643+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:42.990 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:42.643+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:42.990 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:42.643+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:42.991 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:42.644+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:43.255 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:42.989+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:43.255 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:42.989+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:43.255 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:42.989+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:43.255 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:42.990+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:43.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:43.254+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:43.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:43.254+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:43.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:43.254+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:43.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:43.255+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:43.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:43.612+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:43.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:43.612+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:43.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:43.612+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:43.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:43.613+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:44.250 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:43.974+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:44.250 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:43.974+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:44.250 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:43.974+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:44.250 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:43.976+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:44.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:44.248+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:44.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:44.248+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:44.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:44.248+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:44.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:44.249+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:44.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:44.592+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:44.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:44.592+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:44.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:44.592+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:44.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:44.593+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:45.284 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:44.951+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:45.284 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:44.951+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:45.284 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:44.951+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:45.284 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:44.953+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:45.558 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:45.283+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:45.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:45.283+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:45.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:45.283+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:45.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:45.284+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:45.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:45.557+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:45.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:45.557+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:45.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:45.557+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:45.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:45.558+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:46.241 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:42:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:42:45.864+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25567 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:42:46.241 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:45.992+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:46.241 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:45.992+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:46.241 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:45.992+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:46.241 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:45.993+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:46.527 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:46.240+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:46.527 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:46.240+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:46.527 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:46.240+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:46.528 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:46.240+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:46.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:46.526+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:46.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:46.526+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:46.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:46.526+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:46.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:46.527+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:47.285 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:46.949+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:47.285 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:46.949+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:47.285 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:46.949+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:47.286 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:46.950+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:47.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:47.284+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:47.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:47.284+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:47.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:47.284+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:47.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:47.285+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:47.518+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:47.518+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:47.518+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:47.519+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:48.270 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:47.910+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:48.270 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:47.910+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:48.270 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:47.910+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:48.270 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:47.911+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:48.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:48.484+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:48.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:48.484+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:48.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:48.484+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:48.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:48.485+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:48.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:48.269+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:48.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:48.269+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:48.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:48.269+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:48.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:48.270+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:49.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:48.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:49.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:48.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:49.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:48.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:49.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:48.958+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:49.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:49.244+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:49.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:49.244+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:49.244+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:49.245+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:49.462+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:49.462+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:49.462+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:49.463+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:50.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:49.947+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:50.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:49.947+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:50.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:49.947+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:50.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:49.948+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:50.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:50.201+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:50.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:50.201+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:50.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:50.201+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:50.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:50.202+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:50.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:50.469+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:50.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:50.469+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:50.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:50.469+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:50.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:50.470+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:51.165 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:42:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:42:50.866+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25658 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:42:51.165 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:50.912+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:51.165 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:50.912+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:51.165 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:50.912+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:51.165 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:50.913+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:51.506 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:51.164+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:51.507 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:51.164+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:51.507 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:51.164+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:51.507 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:51.165+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:51.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:51.505+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:51.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:51.505+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:51.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:51.505+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:51.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:51.506+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:52.188 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:51.911+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:52.189 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:51.911+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:52.189 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:51.911+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:52.189 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:51.912+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:52.497 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:52.187+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:52.497 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:52.187+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:52.497 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:52.187+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:52.497 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:52.188+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:52.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:52.496+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:52.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:52.496+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:52.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:52.496+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:52.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:52.497+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:53.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:52.932+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:53.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:52.932+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:53.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:52.932+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:53.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:52.933+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:53.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:53.164+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:53.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:53.164+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:53.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:53.164+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:53.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:53.165+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:53.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:53.479+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:53.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:53.479+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:53.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:53.479+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:53.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:53.480+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:54.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:54.187+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:54.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:54.187+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:54.187+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:54.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:54.188+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:54.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:53.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:54.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:53.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:54.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:53.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:54.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:53.958+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:54.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:54.502+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:54.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:54.502+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:54.502+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:54.502+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:55.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:55.181+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:55.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:55.181+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:55.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:55.181+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:55.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:55.182+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:55.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:54.980+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:55.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:54.980+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:55.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:54.980+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:55.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:54.981+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:55.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:55.539+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:55.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:55.539+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:55.539+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:55.540+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:56.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:42:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:42:55.869+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25749 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:42:56.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:55.975+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:56.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:55.975+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:56.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:55.975+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:56.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:55.975+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:56.571 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:56.197+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:56.572 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:56.197+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:56.572 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:56.197+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:56.572 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:56.198+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:56.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:56.571+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:56.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:56.571+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:56.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:56.571+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:56.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:56.571+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:57.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:57.168+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:57.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:57.168+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:57.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:57.168+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:57.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:57.169+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:57.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:56.997+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:57.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:56.997+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:57.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:56.997+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:57.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:56.998+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:57.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:57.565+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:57.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:57.565+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:57.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:57.565+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:57.566+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:58.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:58.141+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:58.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:58.141+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:58.141+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:58.142+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:57.994+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:57.994+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:57.994+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:57.996+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:58.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:58.563+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:58.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:58.563+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:58.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:58.563+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:58.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:58.563+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:59.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:59.191+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:59.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:59.191+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:59.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:59.191+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:42:59.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:42:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:42:59.192+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:42:59.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:58.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:59.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:58.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:59.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:58.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:42:59.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:42:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:42:58.958+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:42:59.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:59.553+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:59.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:59.553+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:59.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:59.553+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:42:59.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:42:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:42:59.553+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:00.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:00.176+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:00.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:00.176+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:00.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:00.176+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:00.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:00.177+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:00.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:00.001+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:00.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:00.001+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:00.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:00.001+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:00.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:00.002+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:00.873 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:00.597+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:00.873 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:00.597+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:00.873 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:00.597+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:00.873 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:00.597+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:01.156 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:43:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:43:00.872+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25839 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:43:01.156 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:00.999+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:01.156 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:00.999+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:01.156 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:00.999+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:01.156 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:00.999+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:01.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:01.198+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:01.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:01.198+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:01.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:01.198+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:01.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:01.199+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:01.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:01.572+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:01.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:01.572+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:01.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:01.572+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:01.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:01.573+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:02.226 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:01.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:02.226 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:01.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:02.226 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:01.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:02.226 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:01.958+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:02.543 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:02.225+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:02.543 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:02.225+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:02.544 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:02.225+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:02.544 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:02.226+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:02.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:02.542+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:02.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:02.542+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:02.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:02.542+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:02.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:02.543+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:03.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:02.950+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:03.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:02.950+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:02.950+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:02.951+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:03.190+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:03.190+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:03.190+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:03.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:03.191+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:03.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:03.558+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:03.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:03.558+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:03.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:03.558+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:03.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:03.559+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:04.228 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:03.955+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:04.228 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:03.955+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:04.228 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:03.955+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:04.228 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:03.956+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:04.585 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:04.241+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:04.585 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:04.241+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:04.585 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:04.241+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:04.585 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:04.242+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:04.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:04.584+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:04.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:04.584+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:04.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:04.584+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:04.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:04.585+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:05.235 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:04.980+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:05.235 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:04.980+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:05.236 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:04.980+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:05.236 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:04.981+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:05.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:05.234+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:05.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:05.234+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:05.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:05.234+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:05.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:05.235+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:05.876 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:05.622+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:05.876 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:05.622+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:05.876 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:05.622+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:05.876 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:05.623+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:06.273 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:43:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:43:05.875+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25930 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:43:06.274 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:06.011+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:06.274 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:06.011+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:06.274 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:06.011+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:06.274 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:06.012+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:06.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:06.273+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:06.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:06.273+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:06.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:06.273+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:06.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:06.274+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:06.992 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:06.624+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:06.992 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:06.624+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:06.992 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:06.624+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:06.992 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:06.625+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:07.300 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:06.991+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:07.301 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:06.991+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:07.301 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:06.991+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:07.301 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:06.992+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:07.317+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:07.317+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:07.317+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:07.318+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:07.969 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:07.620+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:07.969 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:07.620+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:07.969 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:07.620+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:07.969 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:07.621+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:08.296 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:07.968+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:08.297 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:07.968+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:08.297 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:07.968+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:08.297 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:07.969+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:08.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:08.296+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:08.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:08.296+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:08.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:08.296+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:08.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:08.297+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:08.959 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:08.619+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:08.959 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:08.619+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:08.960 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:08.619+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:08.960 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:08.619+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:09.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:08.958+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:09.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:08.958+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:09.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:08.958+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:09.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:08.959+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:09.313+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:09.313+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:09.313+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:09.314+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:09.930 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:09.640+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:09.930 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:09.640+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:09.931 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:09.640+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:09.931 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:09.640+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:10.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:09.929+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:10.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:09.929+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:10.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:09.929+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:10.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:09.930+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:10.276+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:10.276+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:10.276+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:10.276+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:10.889 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:43:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:43:10.878+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26021 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:43:10.889 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:10.633+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:10.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:10.633+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:10.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:10.633+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:10.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:10.634+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:11.317 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:10.942+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:11.318 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:10.942+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:11.318 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:10.942+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:11.318 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:10.942+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:11.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:11.317+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:11.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:11.317+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:11.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:11.317+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:11.317+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:12.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:11.894+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:12.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:11.894+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:11.894+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:11.894+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:12.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:11.671+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:12.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:11.671+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:12.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:11.671+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:12.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:11.672+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:12.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:12.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:12.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:12.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:12.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:12.273+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:12.907 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:12.649+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:12.907 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:12.649+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:12.907 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:12.649+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:12.907 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:12.650+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:13.298 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:12.906+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:13.298 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:12.906+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:13.298 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:12.906+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:13.298 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:12.907+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:13.299 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:13.297+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:13.299 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:13.297+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:13.299 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:13.297+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:13.299 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:13.298+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:13.945 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:13.643+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:13.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:13.643+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:13.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:13.643+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:13.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:13.644+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:14.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:13.944+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:14.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:13.944+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:14.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:13.944+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:14.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:13.945+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:14.662 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:14.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:14.662 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:14.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:14.662 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:14.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:14.662 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:14.341+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:14.991 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:14.661+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:14.991 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:14.661+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:14.991 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:14.661+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:14.991 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:14.662+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:15.327 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:14.990+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:15.327 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:14.990+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:15.327 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:14.990+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:15.327 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:14.990+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:15.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:15.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:15.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:15.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:15.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:15.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:15.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:15.327+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:15.965 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:43:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:43:15.881+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26112 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:43:15.965 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:15.658+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:15.965 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:15.658+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:15.965 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:15.658+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:15.965 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:15.658+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:16.307 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:15.964+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:16.307 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:15.964+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:16.307 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:15.964+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:16.307 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:15.965+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:16.306+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:16.306+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:16.306+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:16.307+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:16.678+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:16.678+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:17.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:16.678+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:17.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:16.679+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:16.927+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:16.927+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:16.927+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:16.928+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:17.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:17.284+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:17.284+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:17.284+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:17.285+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:17.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:17.637+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:17.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:17.637+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:17.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:17.637+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:17.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:17.638+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:18.281 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:17.948+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:18.281 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:17.948+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:18.281 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:17.949+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:18.281 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:17.949+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:18.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:18.280+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:18.280+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:18.280+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:18.281+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:18.955 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:18.665+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:18.955 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:18.665+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:18.955 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:18.665+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:18.955 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:18.665+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:19.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:18.954+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:19.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:18.954+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:19.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:18.954+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:19.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:18.955+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:19.281+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:19.281+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:19.281+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:19.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:19.282+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:19.981 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:19.651+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:19.982 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:19.651+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:19.982 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:19.651+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:19.982 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:19.652+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:20.286 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:19.981+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:20.286 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:19.981+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:20.287 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:19.981+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:20.287 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:19.982+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:20.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:20.285+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:20.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:20.285+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:20.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:20.285+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:20.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:20.286+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:20.894 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:43:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:43:20.883+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26202 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:43:20.895 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:20.636+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:20.895 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:20.636+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:20.895 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:20.636+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:20.895 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:20.637+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:21.292 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:20.986+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:21.292 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:20.986+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:21.293 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:20.986+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:21.293 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:20.987+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:21.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:21.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:21.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:21.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:21.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:21.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:21.292+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:22.000 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:21.616+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:22.000 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:21.616+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:22.000 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:21.616+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:22.000 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:21.617+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:22.323 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:21.999+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:22.323 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:21.999+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:22.323 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:21.999+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:22.323 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:22.000+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:22.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:22.322+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:22.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:22.322+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:22.322+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:22.323+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:23.047 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:22.596+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:23.048 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:22.596+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:23.048 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:22.596+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:23.048 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:22.597+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:23.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:23.046+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:23.046+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:23.046+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:23.047+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:23.619 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:23.345+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:23.619 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:23.345+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:23.619 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:23.345+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:23.620 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:23.346+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:23.620 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:23.560+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:23.620 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:23.561+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:23.620 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:23.561+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:23.620 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:23.561+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:24.314 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:24.050+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:24.314 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:24.050+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:24.314 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:24.050+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:24.314 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:24.051+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:24.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:24.313+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:24.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:24.313+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:24.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:24.313+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:24.314+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:24.604+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:24.604+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:24.604+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:24.604+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:25.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:25.101+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:25.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:25.101+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:25.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:25.101+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:25.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:25.102+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:25.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:25.271+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:25.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:25.271+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:25.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:25.271+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:25.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:25.272+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:25.888 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:25.615+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:25.888 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:25.615+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:25.888 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:25.615+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:25.888 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:25.615+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:26.257 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:43:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:43:25.886+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26293 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:43:26.258 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:26.083+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:26.258 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:26.083+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:26.258 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:26.083+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:26.258 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:26.084+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:26.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:26.256+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:26.256+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:26.256+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:26.257+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:27.068 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:26.604+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:27.068 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:26.604+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:27.068 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:26.604+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:27.068 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:26.605+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:27.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:27.253+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:27.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:27.253+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:27.253+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:27.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:27.253+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:27.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:27.067+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:27.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:27.067+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:27.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:27.067+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:27.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:27.068+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:28.075 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:27.655+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:28.075 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:27.655+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:28.075 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:27.655+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:28.075 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:27.655+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:28.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:28.273+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:28.273+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:28.273+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:28.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:28.274+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:28.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:28.074+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:28.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:28.074+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:28.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:28.074+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:28.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:28.075+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:29.043 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:28.656+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:29.044 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:28.656+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:29.044 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:28.656+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:29.044 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:28.656+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:29.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:29.230+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:29.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:29.230+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:29.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:29.230+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:29.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:29.230+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:29.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:29.042+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:29.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:29.042+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:29.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:29.042+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:29.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:29.043+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:29.679+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:29.679+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:29.679+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:30.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:29.680+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:30.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:30.268+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:30.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:30.268+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:30.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:30.268+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:30.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:30.269+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:30.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:30.091+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:30.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:30.091+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:30.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:30.091+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:30.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:30.092+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:31.014 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:43:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:43:30.889+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26384 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:43:31.015 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:30.726+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:31.015 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:30.726+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:31.015 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:30.726+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:31.015 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:30.727+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:31.273 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:31.090+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:31.273 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:31.090+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:31.273 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:31.090+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:31.273 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:31.091+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:31.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:31.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:31.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:31.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:31.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:31.272+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:31.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:31.273+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:31.767+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:31.767+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:31.767+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:31.767+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:32.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:32.224+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:32.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:32.224+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:32.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:32.224+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:32.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:32.225+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:32.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:32.110+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:32.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:32.110+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:32.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:32.110+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:32.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:32.110+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:32.759+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:32.759+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:33.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:32.759+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:33.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:32.760+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:33.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:33.233+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:33.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:33.233+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:33.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:33.233+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:33.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:33.234+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:33.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:33.141+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:33.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:33.141+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:33.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:33.141+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:33.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:33.142+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:33.721+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:33.721+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:34.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:33.721+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:34.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:33.722+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:34.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:34.208+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:34.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:34.208+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:34.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:34.208+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:34.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:34.210+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:34.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:34.156+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:34.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:34.156+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:34.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:34.156+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:34.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:34.156+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:34.723+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:35.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:34.723+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:35.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:34.723+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:35.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:34.724+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:35.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:35.209+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:35.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:35.209+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:35.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:35.209+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:35.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:35.210+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:35.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:35.190+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:35.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:35.190+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:35.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:35.190+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:35.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:35.191+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:36.016 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:43:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:43:35.892+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26475 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:43:36.016 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:35.765+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:36.016 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:35.765+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:36.016 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:35.765+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:36.016 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:35.765+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:36.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:36.207+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:36.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:36.207+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:36.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:36.207+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:36.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:36.208+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:36.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:36.233+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:36.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:36.233+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:36.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:36.233+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:36.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:36.234+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:36.716+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:36.716+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:36.716+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:36.716+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:37.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:37.257+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:37.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:37.257+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:37.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:37.257+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:37.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:37.258+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:37.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:37.193+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:37.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:37.193+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:37.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:37.193+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:37.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:37.194+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:38.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:37.726+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:38.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:37.726+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:38.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:37.726+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:38.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:37.727+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:38.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:38.252+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:38.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:38.252+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:38.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:38.252+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:38.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:38.253+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:38.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:38.208+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:38.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:38.208+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:38.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:38.208+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:38.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:38.209+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:39.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:38.700+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:39.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:38.700+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:39.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:38.700+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:38.701+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:39.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:39.298+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:39.298+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:39.298+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:39.298+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:39.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:39.216+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:39.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:39.216+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:39.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:39.216+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:39.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:39.216+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:39.693+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:39.693+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:39.693+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:39.694+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:40.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:40.221+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:40.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:40.221+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:40.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:40.221+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:40.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:40.222+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:40.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:40.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:40.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:40.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:40.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:40.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:40.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:40.341+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:41.021 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:43:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:43:40.895+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26565 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:43:41.021 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:40.714+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:41.022 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:40.714+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:41.022 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:40.714+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:41.022 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:40.714+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:41.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:41.226+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:41.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:41.226+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:41.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:41.226+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:41.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:41.227+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:41.732 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:41.377+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:41.732 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:41.377+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:41.732 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:41.377+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:41.732 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:41.377+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:42.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:41.761+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:42.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:41.761+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:42.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:41.761+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:42.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:41.762+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:42.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:42.412+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:42.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:42.412+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:42.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:42.412+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:42.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:42.412+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:42.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:42.236+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:42.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:42.236+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:42.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:42.236+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:42.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:42.237+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:43.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:42.751+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:43.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:42.751+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:43.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:42.751+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:43.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:42.751+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:43.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:43.407+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:43.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:43.407+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:43.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:43.407+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:43.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:43.408+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:43.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:43.247+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:43.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:43.247+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:43.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:43.247+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:43.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:43.248+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:44.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:43.736+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:44.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:43.736+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:44.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:43.736+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:44.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:43.737+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:44.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:44.444+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:44.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:44.444+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:44.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:44.444+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:44.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:44.445+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:44.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:44.218+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:44.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:44.218+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:44.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:44.218+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:44.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:44.219+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:45.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:44.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:44.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:44.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:44.776+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:45.494 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:45.195+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:45.494 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:45.195+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:45.494 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:45.195+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:45.494 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:45.196+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:45.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:45.494+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:45.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:45.494+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:45.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:45.494+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:45.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:45.495+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:46.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:43:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:43:45.899+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26656 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:43:46.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:45.785+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:46.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:45.785+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:46.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:45.785+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:46.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:45.786+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:46.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:46.227+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:46.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:46.227+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:46.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:46.227+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:46.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:46.228+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:46.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:46.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:46.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:46.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:46.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:46.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:46.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:46.450+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:47.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:46.807+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:47.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:46.807+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:46.807+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:46.808+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:47.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:47.413+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:47.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:47.413+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:47.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:47.413+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:47.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:47.414+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:47.204+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:47.204+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:47.204+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:47.204+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:47.766+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:47.766+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:47.766+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:47.766+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:48.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:48.415+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:48.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:48.415+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:48.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:48.415+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:48.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:48.416+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:48.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:48.246+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:48.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:48.246+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:48.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:48.246+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:48.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:48.246+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:49.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:48.747+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:49.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:48.747+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:49.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:48.747+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:49.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:48.748+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:49.374+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:49.374+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:49.374+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:49.375+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:49.286+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:49.286+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:49.286+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:49.286+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:49.700+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:49.700+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:49.700+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:49.701+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:50.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:50.388+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:50.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:50.388+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:50.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:50.388+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:50.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:50.389+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:50.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:50.306+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:50.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:50.306+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:50.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:50.306+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:50.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:50.307+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:51.026 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:43:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:43:50.901+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26747 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:43:51.027 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:50.710+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:51.027 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:50.710+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:51.027 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:50.710+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:51.027 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:50.711+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:51.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:51.371+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:51.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:51.371+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:51.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:51.371+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:51.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:51.372+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:51.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:51.283+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:51.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:51.283+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:51.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:51.283+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:51.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:51.284+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:51.661+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:51.661+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:51.661+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:51.662+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:52.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:52.371+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:52.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:52.371+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:52.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:52.371+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:52.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:52.372+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:52.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:52.277+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:52.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:52.277+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:52.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:52.277+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:52.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:52.277+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:52.690+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:52.690+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:52.690+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:52.691+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:53.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:53.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:53.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:53.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:53.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:53.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:53.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:53.341+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:53.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:53.314+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:53.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:53.314+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:53.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:53.314+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:53.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:53.315+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:54.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:53.667+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:54.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:53.667+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:54.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:53.667+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:54.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:53.668+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:54.717 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:54.365+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:54.717 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:54.365+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:54.717 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:54.365+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:54.717 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:54.366+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:54.717 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:54.358+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:54.718 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:54.358+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:54.718 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:54.358+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:54.718 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:54.359+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:55.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:54.716+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:55.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:54.716+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:55.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:54.716+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:55.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:54.717+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:55.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:55.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:55.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:55.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:55.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:55.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:55.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:55.400+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:55.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:55.312+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:55.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:55.312+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:55.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:55.312+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:55.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:55.313+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:56.028 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:43:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:43:55.905+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26834 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:43:56.028 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:55.688+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:56.028 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:55.688+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:56.028 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:55.688+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:56.028 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:55.688+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:56.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:56.392+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:56.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:56.392+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:56.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:56.392+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:56.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:56.393+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:56.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:56.305+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:56.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:56.305+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:56.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:56.305+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:56.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:56.306+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:56.737+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:56.737+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:56.737+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:57.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:56.738+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:57.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:57.291+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:57.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:57.291+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:57.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:57.291+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:57.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:57.291+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:57.382+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:57.382+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:57.382+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:57.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:57.383+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:58.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:57.787+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:58.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:57.787+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:57.787+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:57.787+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:58.361 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:58.286+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:58.361 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:58.286+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:58.361 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:58.286+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:58.361 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:58.287+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:43:58.801 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:58.360+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:58.802 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:58.360+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:58.802 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:58.360+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:58.802 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:58.361+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:59.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:58.800+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:59.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:58.800+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:59.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:58.800+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:43:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:58.801+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:59.407+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:59.407+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:59.407+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:43:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:43:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:43:59.409+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:43:59.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:59.299+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:59.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:59.299+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:59.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:59.299+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:43:59.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:43:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:43:59.300+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:00.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:59.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:59.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:59.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:43:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:43:59.764+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:00.393+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:00.393+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:00.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:00.393+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:00.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:00.394+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:00.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:00.345+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:00.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:00.345+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:00.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:00.345+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:00.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:00.346+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:01.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:44:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:44:00.908+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26924 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:44:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:00.791+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:00.791+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:00.791+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:00.792+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:01.734 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:01.343+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:01.734 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:01.343+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:01.734 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:01.343+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:01.734 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:01.344+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:01.734 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:01.358+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:01.734 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:01.358+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:01.734 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:01.358+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:01.734 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:01.360+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:01.742+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:01.742+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:01.742+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:01.743+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:02.341 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:02.299+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:02.341 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:02.299+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:02.341 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:02.299+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:02.341 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:02.300+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:02.596 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:02.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:02.596 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:02.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:02.596 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:02.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:02.596 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:02.341+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:03.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:02.746+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:03.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:02.746+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:03.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:02.746+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:02.747+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:03.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:03.373+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:03.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:03.373+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:03.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:03.373+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:03.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:03.374+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:03.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:03.320+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:03.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:03.320+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:03.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:03.320+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:03.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:03.321+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:04.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:03.756+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:04.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:03.756+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:04.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:03.756+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:04.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:03.757+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:04.416+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:04.416+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:04.416+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:04.417+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:04.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:04.312+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:04.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:04.312+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:04.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:04.312+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:04.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:04.313+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:05.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:04.799+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:05.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:04.799+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:05.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:04.799+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:05.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:04.800+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:05.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:05.406+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:05.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:05.406+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:05.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:05.406+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:05.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:05.407+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:05.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:05.309+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:05.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:05.309+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:05.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:05.309+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:05.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:05.310+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:05.922 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:44:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:44:05.911+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27015 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:44:05.922 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:05.815+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:05.923 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:05.815+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:05.923 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:05.815+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:05.923 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:05.816+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:06.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:06.422+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:06.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:06.422+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:06.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:06.422+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:06.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:06.423+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:06.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:06.283+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:06.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:06.283+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:06.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:06.283+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:06.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:06.284+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:07.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:06.808+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:06.808+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:06.808+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:06.809+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:07.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:07.305+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:07.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:07.305+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:07.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:07.305+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:07.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:07.306+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:07.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:07.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:07.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:07.455+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:07.849+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:08.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:07.849+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:08.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:07.849+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:08.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:07.850+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:08.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:08.410+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:08.741 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:08.410+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:08.741 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:08.410+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:08.741 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:08.411+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:08.741 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:08.339+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:08.741 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:08.339+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:08.741 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:08.339+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:08.741 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:08.341+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:09.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:08.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:09.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:08.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:09.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:08.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:09.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:08.849+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:09.453+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:09.453+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:09.453+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:09.453+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:09.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:09.347+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:09.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:09.347+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:09.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:09.347+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:09.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:09.347+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:10.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:09.890+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:10.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:09.890+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:10.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:09.890+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:10.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:09.890+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:10.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:10.339+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:10.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:10.339+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:10.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:10.339+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:10.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:10.340+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:10.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:10.439+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:10.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:10.439+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:10.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:10.439+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:10.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:10.440+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:11.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:44:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:44:10.914+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27106 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:44:11.387 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:10.841+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:11.388 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:10.841+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:11.388 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:10.841+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:11.388 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:10.842+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:11.734 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:11.415+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:11.734 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:11.415+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:11.734 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:11.415+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:11.734 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:11.416+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:11.735 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:11.343+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:11.735 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:11.343+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:11.735 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:11.343+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:11.735 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:11.344+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:11.844+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:11.844+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:11.844+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:12.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:11.844+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:12.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:12.355+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:12.355+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:12.355+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:12.356+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:12.440+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:12.440+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:12.440+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:12.441+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:13.314 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:12.879+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:13.315 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:12.879+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:13.315 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:12.879+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:13.315 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:12.880+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:13.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:13.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:13.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:13.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:13.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:13.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:13.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:13.478+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:13.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:13.314+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:13.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:13.314+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:13.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:13.314+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:13.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:13.314+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:14.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:13.844+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:14.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:13.844+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:13.844+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:13.845+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:14.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:14.526+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:14.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:14.526+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:14.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:14.526+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:14.527+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:14.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:14.320+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:14.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:14.320+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:14.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:14.320+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:14.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:14.321+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:15.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:14.870+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:15.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:14.870+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:15.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:14.870+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:15.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:14.871+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:15.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:15.349+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:15.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:15.349+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:15.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:15.349+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:15.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:15.350+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:15.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:15.481+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:15.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:15.481+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:15.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:15.481+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:15.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:15.482+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:16.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:44:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:44:15.916+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27197 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:44:16.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:15.909+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:16.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:15.909+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:16.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:15.909+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:16.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:15.910+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:16.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:16.498+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:16.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:16.498+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:16.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:16.498+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:16.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:16.498+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:16.359+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:16.359+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:16.359+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:16.360+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:16.871+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:16.871+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:17.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:16.871+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:17.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:16.872+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:17.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:17.490+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:17.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:17.490+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:17.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:17.490+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:17.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:17.491+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:17.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:17.402+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:17.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:17.402+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:17.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:17.402+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:17.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:17.403+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:17.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:17.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:18.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:17.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:18.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:17.852+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:18.831 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:18.427+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:18.831 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:18.427+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:18.831 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:18.427+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:18.831 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:18.428+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:18.831 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:18.455+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:18.831 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:18.455+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:18.831 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:18.455+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:18.831 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:18.455+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:19.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:18.830+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:19.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:18.830+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:19.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:18.830+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:19.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:18.831+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:19.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:19.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:19.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:19.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:19.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:19.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:19.455+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:19.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:19.477+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:19.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:19.477+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:19.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:19.477+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:19.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:19.477+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:20.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:19.870+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:20.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:19.870+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:20.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:19.870+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:20.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:19.870+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:20.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:20.457+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:20.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:20.457+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:20.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:20.457+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:20.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:20.458+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:20.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:20.460+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:20.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:20.460+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:20.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:20.460+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:20.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:20.461+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:21.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:44:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:44:20.919+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27286 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:44:21.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:20.871+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:21.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:20.871+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:21.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:20.871+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:21.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:20.871+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:21.735 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:21.447+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:21.736 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:21.447+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:21.736 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:21.447+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:21.736 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:21.448+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:21.736 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:21.452+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:21.736 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:21.452+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:21.736 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:21.452+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:21.736 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:21.453+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:21.896+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:21.896+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:21.896+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:22.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:21.897+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:22.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:22.484+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:22.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:22.484+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:22.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:22.484+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:22.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:22.485+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:22.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:22.407+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:22.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:22.407+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:22.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:22.407+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:22.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:22.408+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:23.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:22.927+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:23.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:22.927+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:23.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:22.927+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:23.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:22.927+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:23.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:23.398+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:23.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:23.398+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:23.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:23.398+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:23.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:23.399+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:23.473+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:23.473+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:23.473+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:23.473+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:24.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:23.890+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:24.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:23.890+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:24.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:23.890+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:24.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:23.891+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:24.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:24.518+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:24.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:24.518+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:24.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:24.518+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:24.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:24.519+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:24.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:24.370+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:24.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:24.370+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:24.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:24.370+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:24.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:24.371+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:25.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:24.938+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:25.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:24.938+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:25.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:24.938+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:25.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:24.938+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:25.362+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:25.362+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:25.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:25.362+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:25.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:25.363+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:25.483+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:25.483+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:25.483+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:25.484+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:26.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:44:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:44:25.923+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27378 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:44:26.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:25.895+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:26.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:25.895+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:26.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:25.895+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:26.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:25.896+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:26.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:26.399+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:26.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:26.399+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:26.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:26.399+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:26.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:26.400+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:26.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:26.521+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:26.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:26.521+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:26.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:26.521+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:26.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:26.522+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:26.945+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:26.945+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:26.945+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:26.946+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:27.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:27.530+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:27.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:27.530+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:27.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:27.530+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:27.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:27.531+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:27.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:27.422+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:27.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:27.422+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:27.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:27.422+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:27.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:27.422+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:28.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:27.981+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:28.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:27.981+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:28.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:27.981+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:28.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:27.981+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:28.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:28.387+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:28.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:28.387+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:28.387+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:28.388+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:28.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:28.488+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:28.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:28.488+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:28.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:28.488+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:28.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:28.488+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:29.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:28.941+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:29.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:28.941+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:29.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:28.941+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:29.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:28.942+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:29.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:29.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:29.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:29.434+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:29.480+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:29.480+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:29.480+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:29.480+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:29.908+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:29.908+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:30.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:29.908+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:30.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:29.908+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:30.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:30.460+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:30.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:30.460+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:30.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:30.460+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:30.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:30.461+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:30.497+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:30.497+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:30.497+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:30.498+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:31.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:44:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:44:30.926+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27468 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:44:31.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:30.901+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:31.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:30.901+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:31.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:30.901+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:31.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:30.902+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:31.736 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:31.428+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:31.736 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:31.428+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:31.736 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:31.428+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:31.736 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:31.429+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:31.736 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:31.500+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:31.737 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:31.500+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:31.737 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:31.500+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:31.737 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:31.501+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:31.881+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:31.881+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:31.881+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:31.882+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:32.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:32.530+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:32.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:32.530+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:32.530+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:32.531+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:32.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:32.470+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:32.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:32.470+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:32.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:32.470+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:32.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:32.470+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:33.316 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:32.911+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:33.316 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:32.911+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:33.317 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:32.911+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:33.317 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:32.912+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:33.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:33.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:33.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:33.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:33.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:33.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:33.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:33.480+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:33.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:33.575+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:33.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:33.575+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:33.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:33.575+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:33.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:33.576+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:34.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:33.890+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:34.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:33.890+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:33.890+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:33.891+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:34.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:34.476+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:34.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:34.476+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:34.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:34.476+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:34.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:34.476+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:34.530+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:34.530+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:34.530+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:34.531+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:35.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:34.889+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:35.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:34.889+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:35.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:34.889+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:35.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:34.890+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:35.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:35.520+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:35.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:35.520+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:35.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:35.520+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:35.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:35.521+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:35.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:35.579+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:35.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:35.579+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:35.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:35.579+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:35.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:35.580+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:36.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:44:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:44:35.929+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27560 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:44:36.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:35.880+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:36.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:35.880+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:36.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:35.880+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:36.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:35.880+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:36.504+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:36.504+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:36.504+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:36.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:36.504+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:36.563+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:36.563+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:36.563+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:36.564+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:37.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:36.840+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:37.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:36.840+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:37.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:36.840+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:37.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:36.840+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:37.530+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:37.530+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:37.530+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:37.530+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:37.521+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:37.521+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:37.521+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:37.523+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:37.853+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:37.853+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:37.853+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:37.854+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:38.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:38.528+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:38.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:38.528+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:38.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:38.528+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:38.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:38.528+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:38.513+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:38.513+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:38.513+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:38.514+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:39.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:38.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:39.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:38.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:39.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:38.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:39.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:38.849+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:39.547 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:39.481+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:39.547 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:39.481+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:39.548 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:39.481+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:39.548 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:39.482+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:39.808 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:39.546+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:39.808 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:39.546+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:39.808 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:39.546+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:39.809 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:39.547+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:40.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:39.807+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:40.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:39.807+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:40.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:39.807+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:40.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:39.808+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:40.794 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:40.476+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:40.794 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:40.476+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:40.794 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:40.476+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:40.794 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:40.477+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:40.794 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:40.523+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:40.794 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:40.523+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:40.794 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:40.523+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:40.794 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:40.524+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:41.057 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:44:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:44:40.932+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27649 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:44:41.058 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:40.793+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:41.058 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:40.793+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:41.058 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:40.793+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:41.058 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:40.794+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:41.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:41.493+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:41.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:41.493+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:41.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:41.493+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:41.801 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:41.494+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:41.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:41.496+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:41.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:41.496+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:41.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:41.496+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:41.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:41.497+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:42.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:41.799+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:42.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:41.799+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:42.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:41.799+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:42.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:41.800+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:42.758 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:42.477+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:42.758 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:42.477+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:42.758 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:42.477+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:42.759 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:42.478+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:42.759 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:42.471+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:42.759 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:42.471+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:42.759 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:42.471+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:42.759 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:42.472+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:43.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:42.757+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:43.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:42.757+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:43.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:42.757+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:43.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:42.758+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:43.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:43.502+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:43.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:43.502+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:43.502+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:43.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:43.504+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:43.722+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:43.722+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:43.722+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:43.723+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:43.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:43.480+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:43.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:43.480+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:43.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:43.480+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:43.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:43.481+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:44.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:44.450+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:44.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:44.450+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:44.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:44.450+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:44.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:44.451+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:44.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:44.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:44.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:44.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:44.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:44.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:44.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:44.493+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:45.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:44.720+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:45.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:44.720+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:45.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:44.720+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:44.721+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:45.733 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:45.454+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:45.733 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:45.454+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:45.733 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:45.454+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:45.733 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:45.455+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:45.733 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:45.486+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:45.733 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:45.486+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:45.733 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:45.486+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:45.734 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:45.486+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:46.061 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:44:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:44:45.935+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27741 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:44:46.061 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:45.732+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:46.061 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:45.732+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:46.061 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:45.732+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:46.061 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:45.732+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:46.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:46.503+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:46.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:46.503+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:46.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:46.503+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:46.504+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:46.516+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:46.516+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:46.516+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:46.517+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:46.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:46.730+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:46.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:46.730+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:46.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:46.730+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:46.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:46.731+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:47.777 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:47.548+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:47.777 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:47.548+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:47.778 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:47.548+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:47.778 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:47.548+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:47.778 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:47.487+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:47.778 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:47.487+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:47.778 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:47.487+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:47.778 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:47.488+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:48.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:47.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:48.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:47.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:48.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:47.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:47.777+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:48.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:48.559+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:48.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:48.559+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:48.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:48.559+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:48.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:48.560+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:48.821 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:48.446+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:48.821 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:48.446+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:48.821 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:48.446+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:48.821 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:48.446+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:49.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:48.820+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:49.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:48.820+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:49.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:48.820+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:49.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:48.821+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:49.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:49.450+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:49.450+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:49.450+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:49.451+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:50.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:49.589+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:50.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:49.589+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:49.589+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:49.591+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:49.820+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:49.821+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:49.821+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:49.821+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:50.566 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:50.422+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:50.566 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:50.422+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:50.566 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:50.422+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:50.566 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:50.422+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:50.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:50.773+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:50.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:50.773+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:50.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:50.773+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:50.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:50.774+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:50.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:50.565+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:50.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:50.565+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:50.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:50.565+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:50.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:50.566+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:51.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:44:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:44:50.939+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27831 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:44:51.738 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:51.611+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:51.738 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:51.611+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:51.738 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:51.611+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:51.738 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:51.611+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:51.738 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:51.453+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:51.738 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:51.453+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:51.738 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:51.453+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:51.738 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:51.454+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:51.758+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:51.758+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:51.758+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:51.758+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:52.756 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:52.596+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:52.756 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:52.596+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:52.756 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:52.596+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:52.756 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:52.597+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:52.756 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:52.417+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:52.756 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:52.417+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:52.756 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:52.417+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:52.756 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:52.418+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:52.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:52.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:52.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:52.755+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:53.729 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:53.568+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:53.730 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:53.568+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:53.730 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:53.568+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:53.730 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:53.569+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:53.730 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:53.454+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:53.730 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:53.454+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:53.730 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:53.454+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:53.730 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:53.455+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:54.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:53.728+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:54.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:53.728+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:54.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:53.728+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:54.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:53.729+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:54.738 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:54.546+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:54.739 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:54.546+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:54.739 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:54.546+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:54.739 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:54.547+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:54.739 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:54.442+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:54.739 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:54.442+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:54.739 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:54.442+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:54.739 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:54.442+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:55.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:54.737+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:55.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:54.737+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:55.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:54.737+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:55.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:54.738+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:55.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:55.539+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:55.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:55.539+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:55.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:55.539+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:55.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:55.540+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:55.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:55.400+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:55.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:55.400+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:55.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:55.400+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:55.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:55.400+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:55.954 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:44:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:44:55.942+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27923 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:44:55.954 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:55.700+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:55.955 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:55.700+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:55.955 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:55.700+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:55.955 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:55.701+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:56.679 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:56.396+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:56.680 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:56.396+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:56.680 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:56.396+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:56.680 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:56.397+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:56.680 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:56.555+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:56.680 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:56.555+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:56.680 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:56.555+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:56.680 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:56.556+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:57.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:56.679+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:57.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:56.679+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:57.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:56.679+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:57.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:56.679+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:57.711 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:57.353+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:57.712 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:57.353+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:57.712 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:57.353+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:57.712 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:57.354+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:57.712 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:57.575+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:57.712 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:57.575+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:57.712 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:57.575+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:57.712 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:57.576+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:58.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:57.711+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:58.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:57.711+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:58.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:57.711+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:58.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:57.711+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:58.575 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:58.324+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:58.575 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:58.324+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:58.575 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:58.324+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:58.575 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:58.325+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:58.574+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:58.574+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:58.574+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:58.575+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:58.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:58.694+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:58.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:58.694+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:58.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:58.694+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:58.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:58.695+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:59.555 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:59.375+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:59.555 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:59.375+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:59.555 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:59.375+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:44:59.555 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:44:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:44:59.376+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:44:59.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:59.554+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:59.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:59.554+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:59.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:59.554+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:44:59.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:44:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:44:59.555+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:44:59.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:59.674+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:59.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:59.674+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:59.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:59.674+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:44:59.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:44:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:44:59.674+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:00.539 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:00.408+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:00.539 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:00.408+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:00.539 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:00.408+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:00.539 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:00.409+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:00.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:00.681+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:00.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:00.681+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:00.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:00.681+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:00.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:00.682+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:00.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:00.538+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:00.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:00.538+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:00.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:00.538+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:00.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:00.539+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:01.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:45:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:45:00.945+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28012 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:45:01.720 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:01.549+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:01.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:01.549+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:01.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:01.549+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:01.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:01.550+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:01.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:01.396+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:01.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:01.396+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:01.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:01.396+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:01.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:01.397+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:01.720+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:01.720+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:01.720+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:01.720+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:02.763 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:02.505+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:02.764 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:02.505+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:02.764 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:02.505+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:02.764 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:02.506+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:02.764 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:02.396+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:02.764 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:02.396+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:02.764 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:02.396+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:02.764 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:02.398+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:03.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:02.762+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:03.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:02.762+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:03.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:02.762+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:02.763+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:03.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:03.499+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:03.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:03.499+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:03.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:03.499+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:03.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:03.499+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:03.729 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:03.350+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:03.729 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:03.350+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:03.729 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:03.350+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:03.729 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:03.350+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:04.035 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:03.727+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:04.036 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:03.727+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:04.036 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:03.727+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:04.036 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:03.728+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:04.754 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:04.468+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:04.754 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:04.468+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:04.754 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:04.468+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:04.754 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:04.469+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:04.755 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:04.356+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:04.755 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:04.356+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:04.755 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:04.356+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:04.755 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:04.357+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:05.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:04.754+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:05.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:04.754+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:05.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:04.754+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:05.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:04.755+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:05.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:05.318+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:05.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:05.318+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:05.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:05.318+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:05.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:05.319+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:05.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:05.427+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:05.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:05.427+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:05.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:05.427+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:05.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:05.428+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:06.078 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:45:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:45:05.949+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28104 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:45:06.078 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:05.749+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:06.078 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:05.749+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:06.078 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:05.749+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:06.078 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:05.749+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:06.755 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:06.339+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:06.755 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:06.339+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:06.755 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:06.339+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:06.755 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:06.340+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:06.755 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:06.431+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:06.755 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:06.431+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:06.756 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:06.431+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:06.756 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:06.432+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:07.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:06.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:07.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:06.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:06.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:06.756+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:07.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:07.319+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:07.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:07.319+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:07.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:07.319+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:07.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:07.320+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:07.477+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:07.477+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:07.477+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:07.478+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:08.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:07.771+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:08.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:07.771+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:08.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:07.771+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:07.772+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:08.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:08.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:08.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:08.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:08.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:08.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:08.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:08.507+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:08.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:08.320+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:08.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:08.320+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:08.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:08.320+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:08.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:08.321+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:08.726+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:08.726+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:08.726+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:08.726+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:09.496 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:09.344+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:09.497 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:09.344+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:09.497 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:09.344+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:09.497 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:09.344+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:09.751 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:09.496+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:09.751 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:09.496+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:09.751 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:09.496+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:09.752 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:09.497+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:10.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:09.751+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:10.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:09.751+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:10.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:09.751+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:09.752+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:10.524 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:10.327+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:10.524 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:10.327+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:10.524 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:10.327+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:10.524 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:10.328+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:10.787 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:10.524+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:10.787 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:10.524+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:10.787 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:10.524+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:10.787 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:10.525+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:11.077 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:45:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:45:10.953+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28194 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:45:11.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:10.787+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:11.078 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:10.787+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:11.078 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:10.787+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:11.078 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:10.788+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:11.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:11.531+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:11.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:11.531+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:11.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:11.531+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:11.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:11.532+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:11.740 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:11.349+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:11.740 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:11.349+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:11.740 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:11.349+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:11.740 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:11.350+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:11.824+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:11.824+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:11.824+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:12.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:11.824+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:12.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:12.497+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:12.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:12.497+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:12.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:12.497+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:12.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:12.497+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:12.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:12.312+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:12.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:12.312+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:12.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:12.312+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:12.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:12.313+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:12.821+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:12.821+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:12.821+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:13.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:12.822+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:13.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:13.286+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:13.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:13.286+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:13.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:13.286+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:13.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:13.286+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:13.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:13.490+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:13.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:13.490+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:13.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:13.490+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:13.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:13.491+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:13.802+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:13.802+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:13.802+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:13.803+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:14.539 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:14.256+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:14.539 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:14.256+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:14.539 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:14.256+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:14.539 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:14.257+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:14.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:14.756+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:14.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:14.756+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:14.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:14.756+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:14.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:14.757+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:14.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:14.538+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:14.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:14.538+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:14.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:14.538+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:14.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:14.539+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:15.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:15.272+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:15.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:15.272+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:15.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:15.272+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:15.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:15.273+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:15.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:15.499+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:15.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:15.499+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:15.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:15.499+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:15.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:15.500+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:15.980 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:45:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:45:15.955+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28286 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:45:15.980 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:15.727+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:15.980 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:15.727+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:15.980 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:15.727+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:15.980 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:15.728+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:16.325 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:16.228+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:16.325 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:16.228+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:16.325 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:16.228+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:16.325 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:16.229+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:16.764 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:16.468+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:16.764 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:16.468+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:16.764 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:16.468+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:16.764 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:16.469+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:16.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:16.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:16.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:16.764+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:17.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:17.492+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:17.492+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:17.492+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:17.493+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:17.245+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:17.245+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:17.245+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:17.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:17.246+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:18.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:17.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:18.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:17.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:18.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:17.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:18.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:17.795+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:18.514 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:18.214+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:18.514 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:18.214+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:18.514 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:18.214+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:18.515 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:18.215+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:18.778 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:18.513+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:18.778 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:18.513+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:18.778 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:18.513+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:18.779 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:18.514+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:19.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:18.777+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:19.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:18.777+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:19.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:18.777+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:19.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:18.778+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:19.497 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:19.225+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:19.497 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:19.225+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:19.497 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:19.225+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:19.497 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:19.226+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:19.792 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:19.496+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:19.792 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:19.496+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:19.792 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:19.496+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:19.792 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:19.497+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:20.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:19.791+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:20.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:19.791+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:20.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:19.791+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:20.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:19.791+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:20.469 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:20.211+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:20.469 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:20.211+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:20.469 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:20.211+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:20.469 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:20.212+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:20.817 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:20.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:20.817 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:20.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:20.817 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:20.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:20.817 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:20.468+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:21.083 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:45:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:45:20.958+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28375 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:45:21.083 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:20.816+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:21.083 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:20.816+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:21.083 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:20.816+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:21.083 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:20.817+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:21.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:21.203+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:21.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:21.203+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:21.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:21.203+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:21.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:21.204+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:21.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:21.435+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:21.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:21.435+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:21.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:21.435+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:21.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:21.435+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:21.814+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:22.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:21.814+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:22.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:21.814+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:22.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:21.814+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:22.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:22.419+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:22.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:22.419+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:22.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:22.419+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:22.420+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:22.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:22.239+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:22.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:22.239+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:22.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:22.239+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:22.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:22.240+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:22.764+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:22.764+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:22.764+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:23.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:22.765+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:23.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:23.451+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:23.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:23.451+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:23.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:23.451+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:23.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:23.452+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:23.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:23.241+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:23.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:23.241+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:23.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:23.241+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:23.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:23.242+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:24.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:23.754+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:24.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:23.754+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:23.754+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:23.754+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:24.483 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:24.199+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:24.483 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:24.199+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:24.483 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:24.199+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:24.483 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:24.200+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:24.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:24.482+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:24.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:24.482+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:24.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:24.482+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:24.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:24.483+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:24.785+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:24.785+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:24.785+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:24.786+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:25.488 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:25.151+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:25.488 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:25.151+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:25.488 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:25.151+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:25.488 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:25.151+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:25.802 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:25.487+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:25.802 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:25.487+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:25.802 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:25.487+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:25.802 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:25.488+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:26.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:45:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:45:25.961+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28467 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:45:26.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:25.801+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:26.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:25.801+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:26.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:25.801+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:26.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:25.802+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:26.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:26.164+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:26.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:26.164+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:26.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:26.164+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:26.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:26.165+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:26.813 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:26.517+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:26.813 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:26.517+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:26.813 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:26.517+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:26.813 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:26.518+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:27.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:26.812+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:27.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:26.812+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:27.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:26.812+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:27.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:26.813+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:27.489 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:27.135+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:27.489 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:27.135+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:27.489 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:27.135+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:27.489 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:27.136+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:27.788 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:27.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:27.788 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:27.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:27.788 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:27.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:27.788 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:27.489+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:28.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:27.787+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:27.787+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:27.787+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:27.787+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:28.447 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:28.183+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:28.447 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:28.183+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:28.447 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:28.183+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:28.447 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:28.184+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:28.761 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:28.446+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:28.761 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:28.446+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:28.761 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:28.446+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:28.761 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:28.447+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:28.760+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:28.760+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:28.760+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:28.761+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:29.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:29.181+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:29.477 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:29.181+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:29.477 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:29.181+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:29.477 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:29.182+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:29.759 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:29.476+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:29.759 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:29.476+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:29.759 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:29.476+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:29.759 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:29.476+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:29.758+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:29.758+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:29.758+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:29.759+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:30.457 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:30.185+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:30.457 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:30.185+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:30.457 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:30.185+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:30.457 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:30.186+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:30.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:30.456+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:30.796 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:30.456+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:30.796 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:30.456+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:30.796 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:30.457+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:31.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:45:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:45:30.964+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28557 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:45:31.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:30.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:31.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:30.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:31.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:30.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:31.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:30.795+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:31.484 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:31.169+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:31.484 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:31.169+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:31.484 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:31.169+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:31.484 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:31.169+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:31.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:31.483+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:31.741 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:31.483+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:31.741 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:31.483+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:31.741 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:31.484+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:31.781+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:31.781+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:31.781+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:31.782+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:32.501 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:32.152+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:32.501 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:32.152+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:32.501 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:32.152+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:32.502 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:32.152+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:32.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:32.500+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:32.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:32.500+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:32.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:32.500+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:32.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:32.501+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:32.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:32.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:32.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:32.795+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:33.506 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:33.185+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:33.507 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:33.185+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:33.507 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:33.185+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:33.507 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:33.186+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:33.771 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:33.505+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:33.772 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:33.505+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:33.772 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:33.505+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:33.772 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:33.507+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:33.771+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:33.771+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:33.771+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:33.771+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:34.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:34.167+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:34.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:34.167+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:34.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:34.167+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:34.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:34.168+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:34.757 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:34.549+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:34.757 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:34.549+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:34.757 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:34.549+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:34.757 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:34.550+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:34.813+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:34.813+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:34.813+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:34.813+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:35.466 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:35.121+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:35.466 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:35.122+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:35.466 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:35.122+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:35.466 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:35.123+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:35.803 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:35.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:35.804 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:35.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:35.804 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:35.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:35.804 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:35.568+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:36.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:45:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:45:35.968+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28649 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:45:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:35.803+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:35.803+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:35.803+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:35.803+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:36.534 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:36.108+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:36.534 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:36.108+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:36.534 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:36.108+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:36.535 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:36.109+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:36.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:36.533+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:36.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:36.533+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:36.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:36.533+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:36.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:36.535+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:36.796+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:36.796+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:36.796+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:36.797+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:37.553 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:37.104+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:37.553 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:37.104+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:37.553 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:37.104+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:37.554 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:37.105+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:37.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:37.552+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:37.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:37.552+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:37.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:37.552+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:37.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:37.553+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:38.080 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:37.824+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:38.080 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:37.824+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:38.080 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:37.824+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:38.080 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:37.825+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:38.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:38.079+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:38.379 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:38.079+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:38.379 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:38.079+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:38.379 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:38.080+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:38.818 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:38.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:38.818 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:38.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:38.818 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:38.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:38.818 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:38.568+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:39.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:39.031+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:39.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:39.031+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:39.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:39.031+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:39.032+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:38.817+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:38.817+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:38.817+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:38.818+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:39.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:39.578+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:39.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:39.578+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:39.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:39.578+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:39.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:39.579+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:39.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:39.770+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:39.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:39.770+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:39.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:39.770+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:39.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:39.771+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:40.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:40.051+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:40.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:40.051+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:40.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:40.051+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:40.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:40.052+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:40.900 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:40.601+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:40.900 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:40.601+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:40.900 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:40.601+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:40.900 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:40.602+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:40.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:40.741+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:40.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:40.741+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:40.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:40.741+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:40.901 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:40.742+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:41.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:45:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:45:40.971+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28738 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:45:41.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:41.091+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:41.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:41.091+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:41.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:41.091+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:41.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:41.091+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:42.067 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:41.598+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:42.067 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:41.598+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:42.067 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:41.598+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:42.067 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:41.599+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:42.067 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:41.749+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:42.067 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:41.749+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:42.067 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:41.749+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:42.067 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:41.750+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:42.066+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:42.066+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:42.066+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:42.067+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:43.085 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:42.621+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:43.085 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:42.622+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:43.085 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:42.622+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:43.085 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:42.622+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:43.086 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:42.773+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:43.086 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:42.773+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:43.086 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:42.773+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:43.086 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:42.774+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:43.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:43.084+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:43.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:43.084+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:43.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:43.084+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:43.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:43.085+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:43.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:43.574+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:43.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:43.574+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:43.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:43.574+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:43.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:43.574+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:43.737+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:43.737+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:43.737+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:43.737+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:44.093+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:44.093+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:44.093+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:44.094+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:44.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:44.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:44.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:44.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:44.567+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:44.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:44.759+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:44.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:44.759+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:44.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:44.759+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:44.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:44.760+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:45.549 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:45.120+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:45.549 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:45.120+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:45.549 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:45.120+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:45.549 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:45.120+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:45.823 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:45.751+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:45.823 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:45.751+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:45.823 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:45.751+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:45.823 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:45.752+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:45.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:45.548+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:45.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:45.548+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:45.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:45.548+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:45.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:45.549+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:46.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:45:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:45:45.974+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28829 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:45:46.523 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:46.096+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:46.523 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:46.096+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:46.523 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:46.096+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:46.523 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:46.097+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:46.798 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:46.522+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:46.798 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:46.522+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:46.798 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:46.522+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:46.798 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:46.523+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:47.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:46.797+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:47.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:46.797+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:47.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:46.797+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:47.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:46.797+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:47.501 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:47.141+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:47.501 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:47.141+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:47.502 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:47.141+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:47.502 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:47.142+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:47.755 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:47.500+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:47.756 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:47.500+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:47.756 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:47.500+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:47.756 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:47.502+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:48.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:47.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:48.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:47.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:48.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:47.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:48.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:47.755+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:48.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:48.138+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:48.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:48.138+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:48.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:48.138+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:48.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:48.138+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:48.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:48.789+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:48.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:48.790+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:48.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:48.790+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:48.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:48.790+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:48.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:48.547+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:48.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:48.547+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:48.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:48.547+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:48.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:48.548+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:49.542 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:49.187+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:49.542 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:49.187+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:49.542 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:49.187+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:49.542 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:49.188+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:49.810 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:49.541+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:49.811 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:49.541+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:49.811 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:49.541+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:49.811 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:49.542+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:49.810+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:49.810+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:49.810+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:49.810+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:50.518 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:50.172+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:50.519 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:50.172+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:50.519 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:50.172+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:50.519 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:50.172+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:50.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:50.518+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:50.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:50.518+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:50.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:50.518+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:50.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:50.519+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:51.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:45:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:45:50.977+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28918 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:45:51.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:50.823+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:50.823+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:50.823+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:50.824+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:51.509 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:51.190+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:51.509 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:51.190+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:51.509 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:51.190+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:51.509 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:51.191+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:51.807 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:51.508+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:51.807 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:51.508+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:51.807 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:51.508+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:51.808 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:51.509+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:51.806+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:51.806+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:51.806+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:51.807+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:52.530 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:52.142+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:52.531 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:52.142+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:52.531 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:52.142+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:52.531 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:52.143+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:52.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:52.530+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:52.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:52.530+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:52.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:52.530+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:52.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:52.531+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:52.824+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:52.824+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:52.824+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:52.825+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:53.507 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:53.165+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:53.508 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:53.165+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:53.508 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:53.165+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:53.508 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:53.165+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:53.810 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:53.505+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:53.810 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:53.505+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:53.810 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:53.505+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:53.811 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:53.505+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:54.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:53.809+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:54.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:53.809+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:54.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:53.809+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:54.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:53.810+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:54.463 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:54.156+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:54.463 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:54.156+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:54.463 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:54.156+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:54.463 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:54.157+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:54.730 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:54.462+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:54.731 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:54.462+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:54.731 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:54.462+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:54.731 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:54.463+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:54.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-19T03:45:54.729+0000 7f85fa163700 0 monclient(hunting): authenticate timed out after 300 2024-03-19T03:45:54.731 INFO:teuthology.orchestra.run.smithi059.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-19T03:45:55.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:54.838+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:55.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:54.838+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:55.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:54.838+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:55.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:54.839+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:55.125+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:55.125+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:55.125+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:55.126+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:55.706 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:55.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:55.706 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:55.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:55.706 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:55.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:55.706 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:55.450+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:55.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:45:55.731 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephadm.py", line 1423, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephadm.py", line 97, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi059 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -e sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 -- bash -c 'ceph orch ps'" 2024-03-19T03:45:55.933 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=89e728a4f41d4131af0d8a9b908bce0c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephadm.py", line 1423, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephadm.py", line 97, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi059 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -e sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 -- bash -c 'ceph orch ps'" 2024-03-19T03:45:55.937 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-19T03:45:55.977 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-19T03:45:55.977 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-19T03:45:55.977 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T03:45:55.982 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:55.807+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:55.982 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:55.807+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:55.983 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:55.807+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:55.983 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:55.807+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:56.003 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-19T03:45:56.003 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T03:45:56.059 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:45:56.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:45:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:45:55.981+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29010 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:45:56.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:56.130+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:56.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:56.130+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:56.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:56.130+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:56.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:56.131+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:56.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:56.412+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:56.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:56.412+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:56.784 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:56.412+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:56.784 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:56.413+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:56.782+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:57.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:56.782+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:57.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:56.782+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:57.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:56.783+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:57.402 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:57.112+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:57.402 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:57.112+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:57.402 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:57.112+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:57.402 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:57.113+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:57.812 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:57.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:57.812 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:57.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:57.812 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:57.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:57.812 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:57.403+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:58.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:57.811+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:58.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:57.811+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:58.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:57.811+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:57.812+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:58.389 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:58.097+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:58.390 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:58.097+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:58.390 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:58.097+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:58.390 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:58.098+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:58.779 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:58.389+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:58.779 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:58.389+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:58.780 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:58.389+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:58.780 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:58.389+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:59.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:58.778+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:58.779+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:58.779+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:45:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:58.779+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:45:59.411 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:59.140+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:59.412 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:59.140+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:59.412 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:59.140+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:45:59.412 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:45:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:45:59.141+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:45:59.806 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:59.410+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:59.806 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:59.410+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:59.806 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:59.410+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:45:59.806 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:45:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:45:59.411+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:00.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:59.805+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:00.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:59.805+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:59.805+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:45:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:45:59.805+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:00.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:00.167+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:00.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:00.167+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:00.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:00.167+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:00.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:00.168+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:00.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:00.370+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:00.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:00.370+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:00.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:00.370+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:00.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:00.370+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:01.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:46:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:46:00.984+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29099 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:46:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:00.810+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:00.810+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:00.810+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:00.811+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:01.420 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:01.188+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:01.420 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:01.188+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:01.420 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:01.188+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:01.420 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:01.189+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:01.743 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:01.419+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:01.743 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:01.419+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:01.743 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:01.419+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:01.743 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:01.420+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:01.810+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:01.810+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:01.810+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:01.811+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:02.404 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:02.212+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:02.404 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:02.212+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:02.404 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:02.212+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:02.404 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:02.213+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:02.830 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:02.409+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:02.831 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:02.409+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:02.831 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:02.409+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:02.831 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:02.410+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:03.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:02.829+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:02.829+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:02.829+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:02.830+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:03.363 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:03.256+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:03.363 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:03.256+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:03.363 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:03.256+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:03.363 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:03.257+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:03.818 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:03.362+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:03.819 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:03.362+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:03.819 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:03.362+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:03.819 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:03.363+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:04.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:03.817+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:04.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:03.817+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:04.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:03.817+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:04.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:03.818+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:04.384+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:04.384+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:04.384+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:04.385+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:04.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:04.218+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:04.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:04.218+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:04.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:04.218+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:04.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:04.219+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:05.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:04.779+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:05.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:04.779+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:04.779+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:04.780+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:05.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:05.397+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:05.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:05.397+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:05.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:05.397+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:05.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:05.398+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:05.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:05.267+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:05.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:05.267+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:05.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:05.267+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:05.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:05.267+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:05.989 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:05.731+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:05.989 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:05.731+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:05.989 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:05.731+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:05.989 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:05.731+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:06.252 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:46:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:46:05.988+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29191 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:46:06.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:06.251+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:06.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:06.251+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:06.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:06.251+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:06.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:06.253+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:06.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:06.427+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:06.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:06.427+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:06.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:06.427+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:06.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:06.428+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:07.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:06.707+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:07.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:06.707+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:06.707+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:06.708+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:07.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:07.236+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:07.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:07.236+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:07.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:07.236+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:07.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:07.237+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:07.439+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:07.439+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:07.439+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:07.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:07.440+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:08.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:07.708+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:08.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:07.708+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:07.708+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:07.709+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:08.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:08.441+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:08.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:08.441+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:08.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:08.441+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:08.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:08.442+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:08.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:08.207+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:08.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:08.207+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:08.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:08.207+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:08.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:08.208+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:09.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:08.701+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:09.131 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:08.701+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:09.131 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:08.701+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:09.131 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:08.702+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:09.411+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:09.411+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:09.411+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:09.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:09.412+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:09.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:09.200+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:09.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:09.200+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:09.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:09.200+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:09.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:09.201+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:10.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:09.747+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:09.747+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:09.747+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:09.748+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:10.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:10.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:10.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:10.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:10.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:10.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:10.401+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:10.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:10.251+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:10.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:10.251+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:10.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:10.251+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:10.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:10.251+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:11.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:46:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:46:10.991+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29281 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:46:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:10.766+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:10.766+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:10.766+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:10.767+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:11.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:11.255+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:11.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:11.255+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:11.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:11.255+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:11.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:11.256+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:11.620 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:11.366+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:11.620 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:11.366+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:11.620 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:11.366+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:11.620 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:11.367+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:11.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:11.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:12.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:11.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:12.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:11.795+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:12.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:12.273+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:12.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:12.273+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:12.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:12.273+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:12.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:12.274+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:12.374+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:12.374+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:12.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:12.374+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:12.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:12.374+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:12.822+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:12.822+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:13.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:12.822+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:13.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:12.823+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:13.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:13.241+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:13.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:13.241+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:13.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:13.241+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:13.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:13.241+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:13.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:13.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:13.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:13.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:13.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:13.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:13.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:13.327+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:14.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:13.807+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:13.807+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:13.807+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:13.807+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:14.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:14.270+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:14.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:14.270+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:14.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:14.270+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:14.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:14.270+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:14.296+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:14.296+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:14.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:14.296+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:14.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:14.297+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:15.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:14.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:14.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:14.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:14.777+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:15.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:15.288+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:15.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:15.288+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:15.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:15.288+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:15.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:15.289+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:15.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:15.313+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:15.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:15.313+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:15.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:15.313+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:15.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:15.313+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:16.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:46:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:46:15.995+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29372 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:46:16.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:15.822+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:16.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:15.822+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:16.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:15.822+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:16.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:15.823+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:16.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:16.270+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:16.270+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:16.270+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:16.272+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:16.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:16.267+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:16.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:16.267+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:16.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:16.267+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:16.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:16.268+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:16.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:17.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:16.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:17.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:16.776+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:17.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:16.777+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:17.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:17.237+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:17.237+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:17.237+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:17.237+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:17.220+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:17.220+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:17.220+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:17.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:17.221+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:18.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:17.779+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:18.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:17.779+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:18.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:17.779+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:18.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:17.780+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:18.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:18.256+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:18.256+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:18.256+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:18.257+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:18.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:18.221+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:18.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:18.221+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:18.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:18.221+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:18.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:18.221+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:19.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:18.787+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:19.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:18.787+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:19.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:18.787+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:19.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:18.788+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:19.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:19.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:19.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:19.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:19.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:19.291+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:19.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:19.213+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:19.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:19.213+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:19.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:19.213+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:19.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:19.214+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:20.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:19.756+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:20.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:19.756+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:20.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:19.756+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:20.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:19.757+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:20.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:20.245+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:20.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:20.245+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:20.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:20.245+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:20.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:20.246+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:20.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:20.237+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:20.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:20.237+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:20.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:20.237+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:20.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:20.237+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:21.010 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:46:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:46:20.998+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29452 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:46:21.010 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:20.752+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:21.010 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:20.752+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:21.011 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:20.752+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:21.011 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:20.753+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:21.284 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:21.253+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:21.285 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:21.253+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:21.285 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:21.253+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:21.285 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:21.253+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:21.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:21.284+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:21.284+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:21.284+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:21.285+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:21.757+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:22.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:21.757+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:22.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:21.757+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:22.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:21.757+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:22.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:22.303+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:22.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:22.303+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:22.303+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:22.304+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:22.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:22.236+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:22.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:22.236+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:22.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:22.236+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:22.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:22.236+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:22.785+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:23.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:22.785+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:23.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:22.785+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:23.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:22.786+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:23.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:23.323+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:23.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:23.323+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:23.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:23.323+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:23.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:23.325+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:23.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:23.274+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:23.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:23.274+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:23.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:23.274+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:23.591 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:23.274+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:24.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:23.830+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:23.830+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:23.830+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:23.830+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:24.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:24.279+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:24.279+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:24.279+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:24.279+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:24.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:24.296+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:24.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:24.296+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:24.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:24.296+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:24.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:24.296+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:24.824+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:24.824+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:24.824+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:24.826+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:25.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:25.277+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:25.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:25.277+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:25.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:25.277+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:25.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:25.278+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:25.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:25.247+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:25.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:25.247+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:25.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:25.247+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:25.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:25.248+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:26.127 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:46:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:46:26.002+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29543 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:46:26.128 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:25.840+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:26.128 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:25.840+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:26.128 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:25.840+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:26.128 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:25.841+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:26.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:26.267+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:26.267+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:26.267+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:26.267+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:26.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:26.315+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:26.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:26.315+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:26.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:26.315+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:26.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:26.316+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:27.269 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:26.879+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:27.270 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:26.879+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:27.270 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:26.879+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:27.270 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:26.880+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:27.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:27.269+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:27.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:27.269+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:27.269+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:27.270+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:27.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:27.288+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:27.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:27.288+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:27.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:27.288+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:27.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:27.289+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:28.256 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:27.897+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:28.256 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:27.897+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:28.256 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:27.897+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:28.256 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:27.898+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:28.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:28.256+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:28.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:28.256+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:28.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:28.256+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:28.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:28.257+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:28.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:28.329+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:28.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:28.329+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:28.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:28.329+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:28.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:28.330+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:29.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:28.867+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:29.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:28.867+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:29.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:28.867+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:29.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:28.868+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:29.300+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:29.300+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:29.300+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:29.301+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:29.292+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:29.292+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:29.292+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:29.293+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:30.208 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:29.893+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:30.208 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:29.893+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:30.208 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:29.893+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:30.208 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:29.894+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:30.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:30.308+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:30.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:30.308+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:30.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:30.308+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:30.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:30.309+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:30.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:30.245+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:30.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:30.245+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:30.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:30.245+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:30.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:30.246+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:31.130 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:46:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:46:31.005+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29633 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:46:31.131 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:30.855+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:31.131 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:30.855+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:31.131 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:30.855+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:31.131 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:30.856+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:31.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:31.353+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:31.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:31.353+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:31.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:31.353+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:31.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:31.354+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:31.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:31.229+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:31.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:31.229+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:31.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:31.229+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:31.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:31.230+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:31.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:31.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:31.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:31.819+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:32.396 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:32.217+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:32.396 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:32.217+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:32.396 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:32.217+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:32.396 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:32.218+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:32.811 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:32.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:32.811 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:32.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:32.811 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:32.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:32.812 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:32.397+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:32.811+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:32.811+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:32.811+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:33.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:32.812+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:33.376 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:33.229+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:33.376 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:33.229+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:33.376 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:33.229+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:33.377 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:33.230+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:33.637 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:33.375+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:33.637 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:33.375+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:33.638 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:33.375+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:33.638 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:33.376+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:33.821+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:33.821+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:34.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:33.821+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:34.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:33.822+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:34.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:34.195+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:34.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:34.195+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:34.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:34.195+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:34.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:34.196+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:34.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:34.426+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:34.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:34.426+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:34.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:34.426+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:34.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:34.427+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:34.821+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:34.821+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:35.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:34.821+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:35.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:34.821+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:35.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:35.455+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:35.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:35.455+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:35.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:35.455+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:35.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:35.456+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:35.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:35.224+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:35.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:35.224+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:35.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:35.224+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:35.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:35.225+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:36.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:46:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:46:36.008+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29725 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:46:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:35.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:35.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:35.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:35.795+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:36.434 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:36.272+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:36.434 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:36.272+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:36.434 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:36.272+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:36.435 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:36.273+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:36.759 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:36.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:36.759 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:36.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:36.760 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:36.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:36.760 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:36.434+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:36.758+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:36.758+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:36.758+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:36.760+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:37.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:37.437+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:37.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:37.437+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:37.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:37.437+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:37.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:37.438+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:37.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:37.294+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:37.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:37.294+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:37.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:37.294+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:37.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:37.295+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:38.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:37.716+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:38.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:37.716+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:38.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:37.716+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:38.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:37.717+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:38.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:38.307+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:38.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:38.307+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:38.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:38.307+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:38.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:38.308+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:38.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:38.389+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:38.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:38.389+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:38.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:38.389+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:38.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:38.390+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:39.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:38.688+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:39.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:38.688+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:38.688+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:38.688+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:39.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:39.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:39.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:39.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:39.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:39.402+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:39.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:39.275+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:39.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:39.275+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:39.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:39.275+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:39.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:39.276+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:40.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:39.721+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:40.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:39.721+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:40.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:39.721+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:39.722+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:40.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:40.451+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:40.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:40.451+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:40.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:40.451+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:40.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:40.451+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:40.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:40.272+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:40.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:40.272+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:40.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:40.272+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:40.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:40.273+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:41.012 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:40.733+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:41.012 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:40.733+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:41.012 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:40.733+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:41.013 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:40.734+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:41.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:46:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:46:41.011+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29814 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:46:41.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:41.261+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:41.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:41.261+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:41.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:41.261+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:41.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:41.262+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:41.745 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:41.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:41.746 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:41.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:41.746 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:41.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:41.746 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:41.467+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:42.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:41.767+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:42.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:41.767+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:42.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:41.767+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:42.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:41.768+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:42.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:42.247+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:42.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:42.247+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:42.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:42.247+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:42.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:42.247+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:42.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:42.494+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:42.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:42.494+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:42.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:42.494+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:42.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:42.495+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:43.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:42.733+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:43.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:42.733+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:43.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:42.733+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:43.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:42.734+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:43.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:43.261+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:43.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:43.261+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:43.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:43.261+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:43.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:43.262+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:43.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:43.455+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:43.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:43.455+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:43.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:43.455+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:43.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:43.455+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:44.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:43.700+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:44.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:43.700+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:44.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:43.700+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:44.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:43.701+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:44.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:44.492+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:44.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:44.492+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:44.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:44.492+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:44.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:44.493+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:44.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:44.303+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:44.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:44.303+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:44.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:44.303+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:44.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:44.304+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:45.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:44.698+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:45.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:44.698+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:45.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:44.698+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:44.699+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:45.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:45.297+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:45.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:45.297+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:45.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:45.297+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:45.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:45.298+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:45.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:45.470+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:45.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:45.470+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:45.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:45.470+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:45.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:45.470+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:46.015 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:45.660+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:46.016 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:45.660+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:46.016 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:45.660+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:46.016 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:45.661+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:46.323 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:46:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:46:46.015+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29906 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:46:46.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:46.322+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:46.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:46.322+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:46.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:46.322+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:46.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:46.324+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:46.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:46.463+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:46.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:46.463+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:46.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:46.463+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:46.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:46.464+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:47.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:46.654+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:47.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:46.654+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:46.654+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:46.654+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:47.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:47.321+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:47.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:47.321+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:47.321+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:47.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:47.322+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:47.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:47.487+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:47.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:47.487+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:47.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:47.487+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:47.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:47.488+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:48.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:47.610+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:47.610+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:47.610+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:47.611+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:48.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:48.482+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:48.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:48.482+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:48.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:48.482+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:48.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:48.483+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:48.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:48.370+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:48.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:48.370+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:48.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:48.370+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:48.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:48.371+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:48.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:48.601+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:48.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:48.601+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:48.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:48.601+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:48.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:48.602+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:49.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:49.330+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:49.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:49.330+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:49.330+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:49.331+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:49.444+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:49.444+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:49.444+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:49.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:49.446+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:49.602+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:49.602+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:49.602+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:49.603+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:50.575 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:50.445+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:50.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:50.445+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:50.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:50.445+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:50.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:50.446+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:50.576 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:50.346+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:50.576 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:50.346+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:50.577 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:50.346+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:14.662322+0000 (oldest deadline 2024-03-19T03:36:34.662322+0000) 2024-03-19T03:46:50.577 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:50.347+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:50.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:50.574+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:50.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:50.574+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:50.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:50.575+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:50.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:50.575+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:51.314 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:46:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:46:51.018+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29996 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:46:51.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:51.314+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:51.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:51.425+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:51.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:51.425+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:51.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:51.425+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:51.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:51.425+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:51.608+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:51.608+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:51.608+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:51.608+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:52.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:52.336+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:52.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:52.402+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:52.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:52.402+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:52.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:52.402+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:52.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:52.402+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:52.653+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:52.653+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:52.653+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:21.573958+0000 (oldest deadline 2024-03-19T03:36:41.573958+0000) 2024-03-19T03:46:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:52.654+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:53.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:53.322+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:53.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:53.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:53.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:53.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:53.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:53.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:53.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:53.396+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:54.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:53.664+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:54.628 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:54.354+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:54.628 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:54.427+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:54.628 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:54.427+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:54.628 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:54.427+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:54.628 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:54.427+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:55.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:54.684+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:55.657 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:55.388+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:55.658 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:55.411+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:55.658 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:55.411+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:55.658 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:55.411+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:55.658 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:55.412+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:56.022 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:55.657+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:56.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:46:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:46:56.022+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30088 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:46:56.664 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:56.375+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:56.665 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:56.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:56.665 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:56.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:56.665 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:56.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:56.665 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:56.433+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:56.664+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:57.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:57.415+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:57.640+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:57.417+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:57.417+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:57.417+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:57.418+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:58.664 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:58.392+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:58.665 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:58.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:58.665 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:58.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:58.665 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:58.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:58.665 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:58.402+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:59.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:58.664+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:46:59.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:46:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:46:59.409+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:46:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:59.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:59.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:59.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:46:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:46:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:46:59.360+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:00.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:46:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:46:59.714+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:00.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:00.457+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:00.380+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:47:00.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:00.380+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:47:00.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:00.380+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:47:00.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:00.380+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:01.027 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:00.698+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:01.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:47:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:47:01.025+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30177 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:47:01.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:01.497+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:01.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:01.379+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:47:01.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:01.379+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:47:01.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:01.379+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:47:01.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:01.380+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:01.722+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:02.691 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:02.456+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:02.691 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:02.416+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:47:02.691 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:02.416+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:47:02.692 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:02.416+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:36:28.907288+0000 (oldest deadline 2024-03-19T03:36:48.907288+0000) 2024-03-19T03:47:02.692 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:02.417+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:03.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:02.690+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:03.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:03.657+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:03.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:03.415+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:03.471+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:04.686 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:04.383+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:04.686 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:04.464+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:05.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:04.685+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:05.719 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:05.410+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:05.719 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:05.501+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:06.029 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:05.718+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:06.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:47:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:47:06.029+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30269 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:47:06.763 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:06.405+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:06.763 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:06.457+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:07.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:06.762+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:07.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:07.434+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:07.783 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:07.415+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:08.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:07.782+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:08.780 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:08.410+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:08.781 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:08.463+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:09.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:08.779+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:09.457+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:09.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:09.472+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:10.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:09.779+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:10.477 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:10.432+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:10.782 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:10.476+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:11.033 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:10.782+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:11.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:47:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:47:11.032+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30359 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:47:11.747 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:11.503+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:11.748 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:11.482+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:11.757+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:12.750 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:12.505+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:12.751 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:12.496+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:12.751 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:12.496+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:12.751 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:12.496+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:12.751 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:12.497+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:12.749+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:13.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:13.471+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:13.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:13.454+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:13.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:13.454+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:13.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:13.454+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:13.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:13.455+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:14.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:13.782+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:14.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:14.511+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:14.783 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:14.498+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:14.783 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:14.498+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:14.783 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:14.498+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:14.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:14.499+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:15.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:14.782+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:15.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:14.782+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:15.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:14.782+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:14.783+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:15.799 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:15.478+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:15.800 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:15.459+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:15.800 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:15.459+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:15.800 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:15.459+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:15.800 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:15.459+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:16.058 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:47:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:47:16.036+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30450 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:47:16.059 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:15.799+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:16.059 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:15.799+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:16.059 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:15.799+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:16.059 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:15.799+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:16.830 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:16.458+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:16.830 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:16.410+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:16.830 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:16.410+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:16.831 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:16.410+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:16.831 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:16.410+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:17.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:16.829+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:17.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:16.829+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:17.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:16.829+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:17.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:16.830+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:17.819 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:17.438+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:17.820 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:17.445+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:17.820 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:17.445+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:17.820 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:17.445+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:17.820 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:17.445+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:18.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:17.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:18.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:17.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:18.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:17.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:18.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:17.819+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:18.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:18.389+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:18.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:18.431+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:18.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:18.431+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:18.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:18.432+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:18.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:18.433+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:19.205 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:18.859+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:19.205 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:18.859+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:19.206 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:18.859+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:19.206 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:18.859+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:19.431+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:19.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:19.421+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:19.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:19.421+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:19.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:19.421+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:19.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:19.422+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:20.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:19.810+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:20.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:19.810+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:20.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:19.810+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:20.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:19.810+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:20.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:20.434+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:20.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:20.410+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:20.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:20.410+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:20.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:20.410+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:20.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:20.411+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:21.040 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:20.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:21.041 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:20.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:21.041 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:20.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:21.041 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:20.849+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:21.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:47:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:47:21.039+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30539 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:47:21.748 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:21.445+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:21.748 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:21.407+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:21.748 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:21.407+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:21.748 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:21.407+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:21.748 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:21.408+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:21.820+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:21.820+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:21.820+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:22.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:21.821+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:22.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:22.456+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:22.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:22.405+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:22.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:22.405+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:22.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:22.405+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:22.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:22.405+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:22.795+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:22.795+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:22.795+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:23.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:22.796+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:23.807 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:23.500+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:23.808 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:23.420+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:23.808 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:23.420+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:23.808 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:23.420+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:23.808 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:23.421+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:24.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:23.807+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:24.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:23.807+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:23.807+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:23.807+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:24.831 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:24.521+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:24.831 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:24.521+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:24.832 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:24.521+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:24.832 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:24.522+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:24.832 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:24.450+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:24.832 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:24.450+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:24.832 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:24.450+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:24.832 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:24.451+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:24.830+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:25.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:24.830+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:25.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:24.830+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:25.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:24.831+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:25.527+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:25.527+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:25.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:25.527+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:25.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:25.528+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:25.471+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:25.471+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:25.471+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:25.472+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:26.167 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:47:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:47:26.043+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30630 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:47:26.168 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:25.872+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:26.168 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:25.872+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:26.168 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:25.872+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:26.168 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:25.872+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:26.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:26.478+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:26.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:26.478+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:26.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:26.478+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:26.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:26.478+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:26.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:26.481+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:26.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:26.481+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:26.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:26.481+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:26.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:26.482+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:26.851+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:26.851+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:26.851+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:26.852+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:27.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:27.452+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:27.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:27.452+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:27.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:27.452+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:27.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:27.452+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:27.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:27.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:27.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:27.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:27.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:27.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:27.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:27.479+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:28.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:27.883+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:28.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:27.883+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:28.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:27.883+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:28.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:27.883+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:28.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:28.501+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:28.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:28.501+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:28.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:28.501+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:28.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:28.503+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:28.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:28.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:28.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:28.489+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:29.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:28.863+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:29.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:28.863+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:29.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:28.863+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:29.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:28.863+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:29.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:29.480+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:29.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:29.480+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:29.480+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:29.481+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:29.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:29.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:29.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:29.479+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:30.211 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:29.841+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:30.211 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:29.841+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:30.212 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:29.841+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:30.212 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:29.841+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:30.493 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:30.450+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:30.493 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:30.450+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:30.494 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:30.450+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:30.494 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:30.451+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:30.825 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:30.493+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:30.825 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:30.493+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:30.825 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:30.493+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:30.825 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:30.493+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:31.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:47:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:47:31.047+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30721 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:47:31.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:30.824+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:30.824+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:30.824+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:30.825+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:31.748 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:31.520+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:31.748 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:31.520+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:31.748 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:31.520+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:31.749 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:31.521+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:31.749 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:31.470+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:31.749 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:31.470+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:31.749 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:31.470+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:31.749 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:31.471+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:31.781+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:31.781+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:31.781+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:31.782+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:32.812 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:32.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:32.812 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:32.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:32.813 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:32.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:32.813 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:32.489+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:32.813 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:32.562+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:32.813 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:32.562+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:32.813 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:32.562+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:32.813 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:32.563+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:32.811+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:32.811+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:33.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:32.811+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:33.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:32.812+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:33.770 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:33.518+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:33.771 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:33.518+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:33.771 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:33.518+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:33.771 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:33.519+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:33.771 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:33.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:33.771 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:33.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:33.772 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:33.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:33.772 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:33.450+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:33.770+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:33.770+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:34.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:33.770+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:34.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:33.770+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:34.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:34.428+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:34.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:34.428+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:34.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:34.428+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:34.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:34.428+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:34.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:34.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:34.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:34.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:34.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:34.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:34.726 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:34.493+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:34.724+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:34.724+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:34.724+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:35.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:34.725+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:35.773 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:35.392+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:35.773 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:35.392+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:35.774 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:35.392+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:35.774 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:35.393+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:35.774 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:35.523+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:35.774 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:35.523+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:35.774 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:35.523+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:35.774 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:35.524+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:36.051 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:35.772+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:36.051 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:35.772+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:36.052 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:35.772+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:36.052 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:35.773+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:36.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:47:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:47:36.050+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30812 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:47:36.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:36.375+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:36.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:36.375+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:36.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:36.375+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:36.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:36.375+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:36.824 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:36.511+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:36.825 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:36.511+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:36.825 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:36.511+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:36.825 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:36.512+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:36.823+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:36.823+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:36.823+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:36.824+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:37.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:37.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:37.397+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:37.397+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:37.510+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:37.510+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:37.510+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:37.511+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:37.859+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:37.859+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:37.859+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:38.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:37.860+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:38.814 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:38.431+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:38.814 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:38.431+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:38.814 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:38.431+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:38.815 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:38.431+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:38.815 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:38.471+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:38.815 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:38.471+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:38.815 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:38.471+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:38.815 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:38.472+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:38.813+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:38.813+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:38.813+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:38.814+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:39.514 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:39.457+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:39.514 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:39.458+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:39.514 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:39.458+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:39.514 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:39.458+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:39.800 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:39.513+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:39.800 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:39.513+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:39.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:39.513+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:39.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:39.514+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:40.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:39.800+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:40.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:39.800+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:40.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:39.800+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:40.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:39.800+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:40.493 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:40.460+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:40.494 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:40.460+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:40.494 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:40.460+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:40.494 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:40.460+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:40.798 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:40.493+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:40.798 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:40.493+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:40.799 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:40.493+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:40.799 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:40.494+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:41.055 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:40.798+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:41.055 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:40.798+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:41.055 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:40.798+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:41.056 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:40.798+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:41.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:47:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:47:41.054+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30902 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:47:41.749 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:41.473+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:41.749 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:41.473+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:41.750 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:41.473+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:41.750 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:41.474+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:41.750 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:41.496+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:41.750 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:41.496+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:41.750 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:41.496+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:41.751 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:41.497+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:42.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:41.806+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:42.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:41.806+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:42.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:41.806+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:42.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:41.807+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:42.761 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:42.493+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:42.761 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:42.493+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:42.761 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:42.493+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:42.761 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:42.494+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:42.761 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:42.453+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:42.762 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:42.453+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:42.762 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:42.453+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:42.762 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:42.454+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:43.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:42.760+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:43.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:42.760+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:43.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:42.760+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:43.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:42.760+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:43.779 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:43.459+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:43.780 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:43.459+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:43.780 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:43.459+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:43.780 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:43.459+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:43.780 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:43.421+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:43.780 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:43.421+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:43.781 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:43.421+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:43.781 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:43.422+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:44.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:43.797+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:44.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:43.797+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:44.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:43.797+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:44.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:43.798+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:44.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:44.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:44.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:44.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:44.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:44.478+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:44.463+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:44.463+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:44.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:44.463+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:44.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:44.464+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:44.842+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:44.842+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:45.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:44.842+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:45.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:44.842+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:45.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:45.511+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:45.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:45.511+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:45.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:45.511+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:45.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:45.512+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:45.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:45.415+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:45.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:45.415+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:45.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:45.415+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:45.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:45.416+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:46.182 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:47:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:47:46.058+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30993 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:47:46.182 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:45.872+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:46.182 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:45.872+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:46.182 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:45.872+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:46.183 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:45.872+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:46.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:46.509+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:46.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:46.509+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:46.509+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:46.510+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:46.449+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:46.449+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:46.449+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:46.451+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:47.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:46.870+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:47.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:46.870+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:47.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:46.870+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:47.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:46.871+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:47.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:47.529+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:47.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:47.529+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:47.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:47.529+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:47.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:47.530+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:47.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:47.497+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:47.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:47.497+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:47.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:47.497+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:47.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:47.498+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:48.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:47.883+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:48.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:47.883+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:47.883+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:47.884+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:48.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:48.461+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:48.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:48.461+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:48.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:48.461+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:48.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:48.462+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:48.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:48.560+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:48.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:48.560+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:48.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:48.560+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:48.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:48.561+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:49.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:48.868+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:49.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:48.868+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:49.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:48.868+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:49.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:48.869+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:49.566 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:49.435+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:49.566 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:49.435+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:49.566 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:49.435+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:49.566 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:49.436+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:49.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:49.566+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:49.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:49.566+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:49.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:49.566+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:49.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:49.567+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:50.211 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:49.863+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:50.212 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:49.863+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:50.212 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:49.863+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:50.212 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:49.863+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:50.818 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:50.480+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:50.818 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:50.480+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:50.818 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:50.480+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:50.818 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:50.481+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:50.819 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:50.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:50.819 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:50.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:50.819 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:50.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:50.819 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:50.596+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:51.076 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:47:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:47:51.061+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31084 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:47:51.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:50.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:51.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:50.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:51.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:50.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:51.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:50.819+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:51.791 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:51.577+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:51.792 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:51.577+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:51.792 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:51.577+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:51.792 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:51.578+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:51.793 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:51.519+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:51.793 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:51.519+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:51.793 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:51.519+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:51.793 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:51.520+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:51.791+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:51.791+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:51.791+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:51.792+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:52.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:52.614+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:52.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:52.614+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:52.798 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:52.614+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:52.798 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:52.615+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:52.798 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:52.495+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:52.798 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:52.495+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:52.798 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:52.495+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:52.798 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:52.496+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:52.797+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:52.797+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:52.797+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:52.798+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:53.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:53.606+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:53.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:53.606+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:53.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:53.606+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:53.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:53.606+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:53.822 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:53.535+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:53.822 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:53.535+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:53.822 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:53.535+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:53.822 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:53.536+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:54.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:53.821+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:54.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:53.821+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:54.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:53.821+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:54.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:53.822+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:54.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:54.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:54.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:54.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:54.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:54.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:54.589+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:54.550+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:54.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:54.550+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:54.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:54.550+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:54.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:54.551+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:54.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:54.786+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:54.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:54.786+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:54.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:54.786+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:54.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:54.787+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:55.573+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:55.573+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:55.573+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:55.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:55.573+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:55.589+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:55.589+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:55.589+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:55.590+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:55.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:55.749+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:55.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:55.749+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:55.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:55.749+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:55.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:55.750+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:56.134 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-19T03:47:56.135 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-03-19T03:47:56.135 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-19T03:47:56.135 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T03:47:56.152 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-19T03:47:56.152 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T03:47:56.209 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:47:56.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:47:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:47:56.066+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31175 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:47:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:56.589+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:56.589+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:56.589+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:56.590+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:56.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:56.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:56.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:56.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:56.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:56.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:56.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:56.572+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:56.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:56.704+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:56.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:56.704+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:56.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:56.704+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:56.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:56.705+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:57.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:57.695+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:57.695+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:57.695+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:57.696+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:57.536+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:57.536+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:57.536+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:57.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:57.537+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:57.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:57.569+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:57.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:57.569+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:57.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:57.569+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:57.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:57.570+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:58.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:58.675+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:58.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:58.675+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:58.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:58.675+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:58.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:58.676+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:58.561+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:58.561+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:58.561+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:58.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:58.562+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:58.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:58.580+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:58.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:58.580+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:58.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:58.580+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:58.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:58.581+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:47:59.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:59.669+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:59.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:59.669+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:59.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:59.669+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:47:59.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:47:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:47:59.670+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:59.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:59.544+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:59.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:59.544+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:59.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:59.544+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:47:59.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:47:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:47:59.545+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:47:59.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:59.564+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:59.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:59.564+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:59.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:59.564+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:47:59.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:47:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:47:59.565+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:00.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:00.550+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:00.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:00.550+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:00.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:00.550+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:00.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:00.551+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:00.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:00.554+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:00.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:00.554+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:00.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:00.554+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:00.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:00.555+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:00.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:00.710+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:00.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:00.710+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:00.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:00.710+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:00.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:00.710+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:01.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:48:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:48:01.070+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31265 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:48:01.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:01.569+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:01.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:01.569+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:01.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:01.569+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:01.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:01.570+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:01.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:01.515+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:01.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:01.515+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:01.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:01.515+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:01.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:01.516+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:01.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:01.665+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:01.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:01.665+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:01.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:01.665+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:01.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:01.666+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:02.790 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:02.497+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:02.791 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:02.497+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:02.791 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:02.497+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:02.791 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:02.498+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:02.791 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:02.590+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:02.791 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:02.590+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:02.792 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:02.590+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:02.792 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:02.591+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:02.792 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:02.624+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:02.792 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:02.624+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:02.792 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:02.624+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:02.792 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:02.624+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:03.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:03.542+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:03.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:03.542+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:03.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:03.542+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:03.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:03.542+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:03.471+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:03.471+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:03.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:03.471+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:03.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:03.471+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:03.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:03.628+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:03.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:03.628+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:03.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:03.628+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:03.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:03.629+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:04.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:04.466+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:04.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:04.466+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:04.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:04.466+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:04.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:04.467+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:04.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:04.592+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:04.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:04.592+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:04.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:04.592+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:04.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:04.593+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:04.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:04.653+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:04.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:04.653+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:04.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:04.653+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:04.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:04.654+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:05.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:05.467+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:05.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:05.467+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:05.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:05.467+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:05.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:05.467+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:05.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:05.638+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:05.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:05.638+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:05.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:05.638+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:05.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:05.638+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:05.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:05.698+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:05.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:05.698+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:05.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:05.698+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:05.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:05.699+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:06.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:48:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:48:06.074+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31356 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:48:06.690 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:06.688+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:06.690 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:06.688+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:06.690 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:06.688+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:06.690 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:06.689+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:06.691 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:06.438+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:06.691 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:06.438+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:06.691 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:06.438+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:06.691 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:06.439+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:06.691 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:06.632+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:06.691 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:06.632+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:06.692 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:06.632+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:06.692 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:06.633+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:07.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:07.430+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:07.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:07.430+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:07.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:07.430+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:07.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:07.431+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:07.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:07.621+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:07.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:07.621+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:07.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:07.621+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:07.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:07.622+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:07.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:07.677+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:07.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:07.677+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:07.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:07.677+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:07.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:07.678+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:08.601 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:08.421+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:08.601 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:08.421+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:08.601 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:08.421+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:08.601 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:08.422+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:09.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:08.600+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:09.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:08.600+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:09.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:08.600+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:09.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:08.601+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:08.660+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:08.660+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:08.660+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:08.661+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:09.566 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:09.446+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:09.567 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:09.446+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:09.567 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:09.446+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:09.567 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:09.447+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:09.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:09.688+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:09.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:09.688+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:09.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:09.688+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:09.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:09.688+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:09.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:09.565+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:09.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:09.565+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:09.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:09.565+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:09.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:09.566+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:10.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:10.479+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:10.479+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:10.479+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:10.479+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:10.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:10.695+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:10.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:10.695+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:10.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:10.695+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:10.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:10.695+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:10.841 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:10.566+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:10.841 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:10.566+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:10.841 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:10.566+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:10.841 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:10.567+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:11.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:48:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:48:11.077+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31447 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:48:11.688 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:11.464+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:11.688 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:11.464+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:11.689 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:11.464+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:11.689 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:11.465+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:11.689 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:11.547+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:11.689 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:11.547+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:11.689 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:11.547+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:11.689 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:11.548+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:11.687+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:11.687+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:11.687+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:12.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:11.688+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:12.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:12.455+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:12.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:12.455+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:12.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:12.455+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:12.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:12.455+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:12.576+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:12.576+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:12.576+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:12.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:12.577+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:12.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:12.684+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:12.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:12.684+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:12.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:12.684+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:12.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:12.685+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:13.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:13.426+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:13.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:13.426+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:13.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:13.426+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:13.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:13.427+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:13.963 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:13.593+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:13.964 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:13.593+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:13.964 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:13.593+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:13.964 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:13.594+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:13.964 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:13.636+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:13.964 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:13.636+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:13.964 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:13.636+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:13.965 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:13.636+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:14.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:14.583+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:14.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:14.583+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:14.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:14.583+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:14.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:14.584+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:14.441+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:14.441+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:14.441+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:14.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:14.442+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:14.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:14.630+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:14.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:14.630+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:14.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:14.630+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:14.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:14.631+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:15.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:15.607+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:15.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:15.607+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:15.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:15.607+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:15.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:15.608+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:15.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:15.656+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:15.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:15.656+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:15.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:15.656+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:15.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:15.657+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:15.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:15.409+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:15.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:15.409+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:15.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:15.409+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:15.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:15.410+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:16.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:48:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:48:16.081+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31537 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:48:16.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:16.576+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:16.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:16.576+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:16.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:16.576+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:16.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:16.576+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:16.386+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:16.386+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:16.386+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:16.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:16.387+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:16.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:16.623+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:16.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:16.623+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:16.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:16.623+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:16.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:16.624+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:17.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:17.397+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:17.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:17.397+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:17.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:17.397+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:17.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:17.398+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:17.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:17.566+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:17.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:17.566+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:17.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:17.566+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:17.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:17.567+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:17.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:17.593+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:17.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:17.593+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:17.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:17.593+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:17.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:17.593+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:18.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:18.589+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:18.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:18.589+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:18.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:18.589+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:18.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:18.590+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:18.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:18.444+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:18.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:18.444+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:18.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:18.444+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:18.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:18.445+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:18.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:18.566+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:18.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:18.566+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:18.842 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:18.566+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:18.842 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:18.567+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:19.558 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:19.398+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:19.558 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:19.398+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:19.559 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:19.398+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:19.559 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:19.399+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:19.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:19.611+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:19.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:19.611+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:19.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:19.611+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:19.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:19.612+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:19.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:19.558+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:19.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:19.558+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:19.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:19.558+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:19.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:19.559+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:20.574 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:20.390+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:20.574 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:20.390+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:20.574 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:20.390+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:20.574 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:20.390+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:20.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:20.573+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:20.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:20.573+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:20.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:20.573+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:20.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:20.574+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:20.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:20.573+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:20.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:20.573+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:20.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:20.573+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:20.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:20.574+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:21.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:48:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:48:21.085+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31627 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:48:21.753 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:21.558+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:21.753 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:21.558+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:21.753 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:21.558+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:21.754 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:21.559+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:21.754 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:21.356+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:21.754 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:21.356+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:21.754 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:21.356+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:21.754 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:21.357+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:21.755 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:21.594+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:21.755 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:21.594+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:21.755 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:21.594+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:21.755 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:21.595+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:22.692 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:22.509+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:22.693 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:22.509+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:22.693 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:22.509+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:22.693 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:22.510+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:22.693 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:22.398+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:22.694 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:22.398+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:22.694 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:22.398+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:22.694 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:22.399+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:22.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:22.632+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:22.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:22.632+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:22.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:22.632+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:22.695 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:22.633+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:23.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:23.495+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:23.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:23.495+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:23.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:23.495+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:23.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:23.496+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:23.389+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:23.389+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:23.389+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:23.390+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:23.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:23.598+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:23.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:23.598+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:23.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:23.598+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:23.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:23.599+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:24.459+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:24.459+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:24.459+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:24.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:24.460+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:24.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:24.357+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:24.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:24.357+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:24.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:24.357+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:24.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:24.357+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:24.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:24.556+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:24.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:24.556+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:24.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:24.556+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:24.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:24.557+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:25.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:25.394+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:25.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:25.394+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:25.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:25.394+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:25.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:25.395+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:25.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:25.443+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:25.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:25.443+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:25.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:25.443+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:25.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:25.444+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:25.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:25.606+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:25.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:25.606+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:25.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:25.606+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:25.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:25.607+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:26.401 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:48:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:48:26.089+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31717 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:48:26.788 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:26.401+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:26.788 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:26.401+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:26.789 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:26.401+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:26.789 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:26.402+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:26.789 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:26.468+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:26.789 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:26.468+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:26.789 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:26.468+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:26.789 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:26.469+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:26.790 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:26.598+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:26.790 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:26.598+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:26.790 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:26.598+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:26.790 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:26.599+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:27.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:27.420+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:27.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:27.420+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:27.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:27.420+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:27.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:27.421+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:27.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:27.434+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:27.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:27.434+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:27.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:27.434+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:27.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:27.435+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:27.601+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:27.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:27.601+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:27.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:27.601+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:27.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:27.601+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:28.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:28.403+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:28.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:28.403+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:28.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:28.403+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:28.404+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:28.604+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:28.604+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:28.604+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:28.604+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:28.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:28.467+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:28.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:28.467+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:28.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:28.467+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:28.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:28.468+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:29.368+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:29.368+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:29.368+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:29.370+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:29.426+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:29.426+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:29.426+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:29.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:29.427+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:29.617+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:29.617+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:29.617+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:29.618+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:30.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:30.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:30.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:30.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:30.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:30.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:30.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:30.362+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:30.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:30.396+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:30.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:30.396+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:30.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:30.396+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:30.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:30.397+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:31.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:30.664+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:31.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:30.664+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:30.664+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:30.665+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:31.377 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:48:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:48:31.093+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31807 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:48:31.655 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:31.404+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:31.655 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:31.404+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:31.655 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:31.404+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:31.655 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:31.404+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:31.656 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:31.376+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:31.656 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:31.376+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:31.656 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:31.376+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:31.656 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:31.377+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:31.654+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:32.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:31.654+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:32.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:31.654+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:32.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:31.655+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:32.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:32.372+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:32.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:32.373+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:32.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:32.373+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:32.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:32.374+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:32.447+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:32.447+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:32.447+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:32.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:32.448+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:32.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:32.614+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:32.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:32.614+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:32.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:32.614+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:32.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:32.615+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:33.644 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:33.362+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:33.644 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:33.362+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:33.644 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:33.362+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:33.645 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:33.363+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:33.645 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:33.455+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:33.645 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:33.455+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:33.645 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:33.455+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:33.645 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:33.456+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:33.643+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:33.643+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:33.643+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:33.644+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:34.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:34.397+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:34.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:34.397+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:34.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:34.397+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:34.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:34.398+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:34.687 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:34.477+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:34.688 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:34.478+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:34.688 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:34.478+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:34.688 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:34.478+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:34.686+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:34.686+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:34.686+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:34.687+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:35.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:35.638+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:35.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:35.638+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:35.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:35.638+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:35.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:35.639+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:35.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:35.441+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:35.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:35.441+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:35.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:35.441+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:35.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:35.441+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:35.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:35.509+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:35.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:35.509+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:35.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:35.509+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:35.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:35.509+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:36.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:48:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:48:36.097+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31898 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:48:36.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:36.551+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:36.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:36.551+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:36.551+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:36.551+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:36.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:36.409+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:36.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:36.409+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:36.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:36.409+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:36.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:36.410+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:36.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:36.653+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:36.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:36.653+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:36.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:36.653+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:36.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:36.654+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:37.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:37.566+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:37.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:37.566+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:37.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:37.566+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:37.567+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:37.426+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:37.426+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:37.426+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:37.427+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:37.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:37.664+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:37.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:37.664+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:37.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:37.664+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:37.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:37.665+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:38.618 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:38.463+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:38.619 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:38.463+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:38.619 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:38.463+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:38.619 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:38.464+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:38.619 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:38.546+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:38.619 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:38.546+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:38.619 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:38.546+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:38.620 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:38.546+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:39.076 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:38.617+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:39.076 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:38.617+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:39.076 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:38.617+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:39.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:38.618+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:39.585 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:39.442+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:39.585 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:39.442+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:39.585 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:39.442+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:39.585 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:39.443+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:39.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:39.595+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:39.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:39.595+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:39.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:39.595+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:39.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:39.596+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:39.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:39.584+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:39.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:39.584+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:39.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:39.584+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:39.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:39.585+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:40.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:40.588+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:40.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:40.588+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:40.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:40.588+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:40.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:40.589+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:40.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:40.552+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:40.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:40.552+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:40.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:40.552+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:40.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:40.553+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:40.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:40.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:40.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:40.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:40.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:40.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:40.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:40.489+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:41.481 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:48:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:48:41.101+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31987 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:48:41.754 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:41.560+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:41.754 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:41.560+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:41.754 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:41.560+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:41.754 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:41.560+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:41.755 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:41.481+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:41.755 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:41.481+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:41.755 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:41.481+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:41.755 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:41.482+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:41.755 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:41.600+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:41.755 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:41.600+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:41.756 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:41.600+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:41.756 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:41.600+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:42.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:42.513+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:42.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:42.513+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:42.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:42.513+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:42.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:42.514+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:42.543+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:42.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:42.543+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:42.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:42.543+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:42.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:42.544+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:42.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:42.624+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:42.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:42.624+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:42.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:42.624+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:42.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:42.624+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:43.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:43.669+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:43.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:43.669+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:43.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:43.669+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:43.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:43.669+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:43.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:43.544+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:43.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:43.544+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:43.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:43.544+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:43.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:43.545+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:43.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:43.565+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:43.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:43.565+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:43.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:43.565+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:43.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:43.565+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:44.542+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:44.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:44.542+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:44.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:44.542+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:44.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:44.543+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:44.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:44.541+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:44.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:44.541+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:44.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:44.541+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:44.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:44.542+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:44.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:44.620+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:44.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:44.620+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:44.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:44.620+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:44.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:44.621+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:45.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:45.658+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:45.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:45.658+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:45.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:45.658+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:45.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:45.658+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:45.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:45.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:45.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:45.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:45.841 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:45.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:45.841 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:45.506+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:45.841 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:45.509+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:45.841 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:45.509+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:45.842 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:45.509+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:45.842 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:45.510+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:46.528 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:48:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:48:46.105+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32077 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:48:46.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:46.553+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:46.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:46.553+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:46.553+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:46.553+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:46.528+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:46.528+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:46.528+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:46.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:46.529+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:46.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:46.634+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:46.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:46.634+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:46.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:46.634+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:46.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:46.635+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:47.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:47.533+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:47.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:47.533+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:47.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:47.534+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:47.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:47.535+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:47.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:47.680+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:47.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:47.680+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:47.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:47.680+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:47.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:47.681+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:47.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:47.514+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:47.841 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:47.514+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:47.841 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:47.514+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:47.841 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:47.515+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:48.582 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:48.531+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:48.582 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:48.531+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:48.583 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:48.531+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:48.583 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:48.532+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:48.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:48.687+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:48.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:48.687+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:48.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:48.687+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:48.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:48.688+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:48.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:48.582+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:48.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:48.582+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:48.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:48.582+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:48.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:48.583+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:49.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:49.660+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:49.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:49.660+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:49.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:49.660+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:49.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:49.661+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:49.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:49.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:49.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:49.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:49.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:49.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:49.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:49.571+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:49.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:49.501+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:49.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:49.501+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:49.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:49.501+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:49.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:49.502+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:50.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:50.462+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:50.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:50.462+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:50.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:50.462+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:50.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:50.463+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:50.589+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:50.590+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:50.590+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:50.591+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:50.659+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:50.659+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:50.659+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:50.660+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:51.363 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:48:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:48:51.109+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32168 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:48:51.640 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:51.442+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:51.641 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:51.442+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:51.641 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:51.442+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:51.641 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:51.443+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:51.641 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:51.593+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:51.641 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:51.593+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:51.642 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:51.593+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:51.642 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:51.594+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:51.640+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:51.640+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:51.640+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:51.641+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:52.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:52.641+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:52.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:52.641+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:52.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:52.641+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:52.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:52.641+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:52.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:52.621+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:52.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:52.621+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:52.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:52.621+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:52.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:52.622+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:52.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:52.443+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:52.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:52.443+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:52.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:52.443+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:52.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:52.444+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:53.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:53.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:53.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:53.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:53.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:53.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:53.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:53.596+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:53.490+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:53.490+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:53.490+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:53.491+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:53.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:53.668+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:53.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:53.668+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:53.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:53.668+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:53.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:53.669+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:54.621+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:54.621+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:54.621+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:54.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:54.622+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:54.673+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:54.673+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:54.673+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:54.673+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:54.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:54.494+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:54.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:54.494+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:54.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:54.494+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:54.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:54.495+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:55.604+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:55.604+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:55.604+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:55.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:55.605+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:55.461+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:55.461+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:55.461+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:55.462+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:55.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:55.633+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:55.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:55.633+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:55.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:55.633+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:55.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:55.633+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:56.461 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:48:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:48:56.112+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32253 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:48:56.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:56.579+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:56.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:56.579+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:56.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:56.579+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:56.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:56.580+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:56.461+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:56.461+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:56.461+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:56.462+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:56.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:56.614+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:56.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:56.614+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:56.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:56.614+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:56.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:56.615+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:57.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:57.491+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:57.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:57.491+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:57.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:57.491+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:57.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:57.492+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:57.625+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:57.625+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:57.625+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:57.626+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:57.618+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:57.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:57.618+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:57.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:57.618+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:57.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:57.619+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:58.672+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:58.672+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:58.672+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:48:58.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:58.673+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:58.453+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:58.453+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:58.453+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:58.454+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:48:58.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:58.655+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:58.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:58.655+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:58.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:58.655+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:48:58.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:58.655+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:48:59.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:59.437+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:59.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:59.437+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:59.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:59.437+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:48:59.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:48:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:48:59.437+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:00.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:59.644+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:00.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:59.644+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:00.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:59.644+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:48:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:48:59.644+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:59.687+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:59.687+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:59.687+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:48:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:48:59.688+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:00.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:00.616+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:00.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:00.616+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:00.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:00.616+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:00.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:00.617+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:00.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:00.656+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:00.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:00.656+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:00.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:00.656+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:00.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:00.656+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:00.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:00.469+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:00.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:00.469+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:00.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:00.469+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:00.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:00.470+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:01.507 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:49:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:49:01.116+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32337 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:49:01.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:01.600+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:01.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:01.600+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:01.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:01.600+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:01.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:01.601+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:01.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:01.695+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:01.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:01.695+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:01.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:01.695+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:01.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:01.696+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:01.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:01.507+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:01.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:01.507+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:01.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:01.507+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:01.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:01.508+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:02.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:02.542+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:02.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:02.542+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:02.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:02.542+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:02.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:02.542+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:02.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:02.575+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:02.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:02.575+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:02.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:02.575+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:02.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:02.575+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:02.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:02.745+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:02.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:02.745+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:02.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:02.745+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:02.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:02.746+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:03.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:03.621+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:03.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:03.621+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:03.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:03.621+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:03.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:03.622+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:03.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:03.745+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:03.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:03.746+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:03.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:03.746+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:03.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:03.746+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:03.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:03.582+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:03.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:03.582+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:03.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:03.582+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:03.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:03.583+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:04.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:04.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:04.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:04.617+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:05.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:04.719+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:05.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:04.719+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:05.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:04.719+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:05.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:04.720+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:04.616+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:04.616+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:05.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:04.616+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:05.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:04.617+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:06.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:05.630+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:06.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:05.630+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:06.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:05.630+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:06.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:05.631+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:05.745+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:05.745+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:05.745+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:05.746+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:06.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:05.626+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:06.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:05.626+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:06.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:05.626+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:06.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:05.627+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:06.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:49:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:49:06.120+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32419 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:49:07.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:06.706+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:07.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:06.706+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:07.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:06.706+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:07.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:06.707+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:06.665+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:06.665+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:07.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:06.665+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:07.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:06.665+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:07.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:06.607+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:07.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:06.607+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:07.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:06.607+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:07.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:06.608+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:08.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:07.751+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:08.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:07.751+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:08.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:07.751+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:08.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:07.751+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:07.691+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:07.691+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:07.691+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:07.691+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:08.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:07.611+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:08.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:07.611+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:08.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:07.611+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:08.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:07.611+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:09.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:08.607+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:09.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:08.607+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:09.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:08.607+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:09.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:08.607+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:09.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:08.721+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:09.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:08.721+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:09.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:08.721+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:09.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:08.721+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:08.714+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:09.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:08.714+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:09.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:08.714+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:09.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:08.715+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:10.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:09.733+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:10.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:09.733+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:10.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:09.733+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:10.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:09.734+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:10.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:09.676+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:10.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:09.676+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:10.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:09.676+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:10.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:09.677+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:09.613+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:10.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:09.613+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:10.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:09.613+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:10.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:09.614+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:10.743+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:10.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:10.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:10.744+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:11.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:10.717+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:11.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:10.717+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:11.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:10.717+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:11.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:10.718+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:10.617+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:11.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:10.617+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:11.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:10.617+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:11.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:10.618+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:11.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:49:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:49:11.125+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32501 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:49:12.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:11.760+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:11.760+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:11.760+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:11.761+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:12.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:11.719+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:12.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:11.719+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:12.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:11.719+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:12.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:11.720+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:12.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:11.665+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:12.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:11.665+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:12.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:11.665+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:12.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:11.666+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:13.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:12.797+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:13.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:12.797+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:13.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:12.797+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:13.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:12.798+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:12.687+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:12.687+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:12.687+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:12.688+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:13.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:12.697+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:13.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:12.697+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:13.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:12.697+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:13.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:12.698+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:13.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:13.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:13.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:13.828+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:14.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:13.681+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:14.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:13.681+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:14.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:13.681+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:14.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:13.682+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:13.742+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:14.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:13.742+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:14.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:13.742+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:14.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:13.742+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:15.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:14.670+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:15.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:14.670+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:15.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:14.670+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:15.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:14.671+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:14.795+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:14.795+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:14.795+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:14.796+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:14.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:15.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:14.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:15.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:14.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:15.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:14.756+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:16.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:15.803+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:16.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:15.803+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:16.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:15.803+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:16.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:15.803+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:15.697+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:15.697+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:15.697+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:15.698+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:16.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:15.789+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:16.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:15.789+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:16.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:15.789+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:16.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:15.790+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:16.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:49:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:49:16.128+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32589 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:49:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:16.808+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:16.808+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:16.808+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:16.809+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:17.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:16.651+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:17.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:16.651+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:17.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:16.651+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:17.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:16.652+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:17.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:16.816+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:17.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:16.816+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:17.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:16.816+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:17.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:16.817+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:18.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:17.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:18.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:17.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:18.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:17.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:18.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:17.852+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:18.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:17.655+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:18.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:17.655+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:18.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:17.655+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:18.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:17.656+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:17.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:18.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:17.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:18.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:17.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:18.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:17.795+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:19.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:18.765+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:19.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:18.765+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:19.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:18.765+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:18.766+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:19.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:18.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:19.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:18.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:19.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:18.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:19.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:18.646+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:19.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:18.887+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:19.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:18.887+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:19.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:18.887+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:19.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:18.887+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:20.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:19.772+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:20.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:19.772+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:20.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:19.772+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:20.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:19.773+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:20.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:19.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:20.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:19.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:20.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:19.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:20.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:19.609+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:20.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:19.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:20.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:19.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:20.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:19.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:20.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:19.849+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:20.813 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:20.560+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:20.814 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:20.560+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:20.814 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:20.560+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:20.814 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:20.560+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:20.814 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:20.779+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:20.814 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:20.779+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:20.815 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:20.779+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:20.815 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:20.779+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:21.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:20.813+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:21.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:20.813+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:21.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:20.813+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:21.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:20.814+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:21.583 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:49:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:49:21.132+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32676 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:49:21.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:21.583+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:21.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:21.583+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:21.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:21.583+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:21.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:21.584+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:21.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:21.813+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:21.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:21.813+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:21.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:21.813+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:21.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:21.814+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:21.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:21.827+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:21.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:21.827+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:21.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:21.827+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:21.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:21.828+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:22.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:22.587+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:22.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:22.587+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:22.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:22.587+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:22.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:22.588+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:23.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:22.841+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:23.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:22.841+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:23.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:22.841+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:22.841+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:23.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:22.859+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:23.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:22.859+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:23.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:22.859+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:23.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:22.860+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:23.875 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:23.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:23.875 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:23.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:23.875 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:23.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:23.876 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:23.609+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:24.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:23.874+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:24.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:23.874+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:24.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:23.874+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:24.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:23.875+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:24.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:23.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:24.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:23.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:24.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:23.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:24.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:23.907+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:24.895 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:24.863+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:24.896 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:24.863+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:24.896 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:24.863+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:24.896 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:24.864+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:24.896 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:24.633+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:24.897 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:24.633+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:24.897 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:24.633+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:24.897 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:24.634+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:25.156 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:24.895+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:25.156 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:24.895+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:25.156 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:24.895+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:25.157 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:24.896+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:25.925 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:25.591+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:25.925 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:25.591+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:25.926 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:25.591+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:25.926 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:25.592+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:25.926 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:25.840+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:25.926 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:25.840+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:25.926 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:25.840+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:25.926 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:25.841+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:26.261 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:49:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:49:26.136+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32767 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:49:26.262 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:25.925+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:26.262 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:25.925+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:26.262 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:25.925+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:26.262 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:25.925+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:26.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:26.543+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:26.801 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:26.543+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:26.801 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:26.543+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:26.801 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:26.544+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:26.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:26.800+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:26.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:26.800+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:26.802 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:26.800+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:27.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:26.801+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:27.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:26.969+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:27.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:26.969+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:27.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:26.969+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:27.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:26.970+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:27.818 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:27.495+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:27.819 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:27.495+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:27.819 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:27.495+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:27.819 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:27.496+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:27.818+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:27.818+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:27.818+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:27.819+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:28.005+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:28.005+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:28.005+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:28.005+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:28.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:28.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:28.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:28.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:28.479+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:29.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:29.026+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:29.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:29.026+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:29.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:29.026+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:29.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:29.027+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:29.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:28.863+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:29.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:28.863+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:29.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:28.863+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:29.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:28.865+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:29.445+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:29.445+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:29.445+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:29.445+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:30.218 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:29.869+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:30.218 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:29.869+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:30.218 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:29.869+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:30.218 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:29.869+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:30.219 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:30.044+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:30.219 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:30.044+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:30.219 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:30.044+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:30.219 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:30.045+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:30.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:30.397+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:30.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:30.397+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:30.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:30.397+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:30.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:30.398+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:31.141 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:30.852+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:31.141 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:30.852+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:31.141 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:30.852+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:31.141 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:30.853+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:31.141 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:31.012+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:31.142 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:31.012+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:31.142 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:31.012+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:31.142 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:31.013+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:31.412 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:49:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:49:31.140+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32858 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:49:31.758 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:31.412+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:31.758 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:31.412+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:31.758 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:31.412+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:31.759 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:31.413+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:32.020 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:31.830+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:32.021 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:31.830+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:32.021 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:31.830+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:32.021 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:31.831+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:32.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:32.020+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:32.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:32.020+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:32.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:32.020+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:32.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:32.021+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:32.802 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:32.407+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:32.802 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:32.407+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:32.802 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:32.407+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:32.802 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:32.407+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:33.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:32.801+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:33.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:32.801+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:33.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:32.801+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:33.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:32.802+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:33.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:33.035+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:33.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:33.035+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:33.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:33.035+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:33.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:33.036+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:33.758 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:33.444+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:33.758 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:33.444+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:33.759 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:33.444+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:33.759 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:33.445+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:34.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:33.758+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:34.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:33.758+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:34.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:33.758+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:34.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:33.759+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:34.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:34.004+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:34.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:34.004+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:34.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:34.004+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:34.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:34.004+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:34.793 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:34.425+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:34.793 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:34.425+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:34.794 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:34.425+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:34.794 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:34.427+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:35.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:34.793+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:35.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:34.793+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:35.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:34.793+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:35.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:34.794+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:35.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:34.977+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:35.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:34.977+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:35.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:34.977+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:35.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:34.977+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:35.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:35.452+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:35.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:35.452+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:35.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:35.452+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:35.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:35.453+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:36.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:35.822+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:36.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:35.822+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:36.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:35.822+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:35.823+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:35.950+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:35.950+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:35.950+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:35.951+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:36.488 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:49:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:49:36.144+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32949 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:49:36.802 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:36.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:36.803 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:36.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:36.803 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:36.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:36.803 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:36.489+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:37.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:36.802+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:37.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:36.802+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:37.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:36.802+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:37.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:36.803+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:36.967+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:36.967+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:36.967+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:36.968+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:37.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:37.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:37.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:37.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:37.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:37.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:37.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:37.506+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:38.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:37.977+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:38.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:37.977+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:38.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:37.977+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:38.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:37.977+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:38.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:37.824+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:38.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:37.824+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:38.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:37.824+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:38.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:37.825+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:38.782 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:38.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:38.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:38.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:38.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:38.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:38.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:38.469+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:39.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:38.782+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:39.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:38.782+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:39.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:38.782+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:38.783+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:38.997+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:38.997+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:38.997+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:38.998+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:39.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:39.425+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:39.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:39.425+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:39.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:39.426+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:39.427+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:39.992 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:39.991+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:39.992 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:39.991+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:39.992 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:39.991+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:39.992 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:39.741+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:39.993 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:39.741+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:39.993 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:39.741+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:39.993 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:39.742+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:40.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:39.992+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:40.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:40.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:40.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:40.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:40.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:40.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:40.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:40.397+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:41.034 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:40.721+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:41.034 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:40.721+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:41.034 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:40.721+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:41.034 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:40.723+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:41.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:49:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:49:41.148+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 33039 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:49:41.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:41.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:41.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:41.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:41.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:41.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:41.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:41.034+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:41.751 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:41.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:41.751 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:41.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:41.751 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:41.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:41.751 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:41.362+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:42.037 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:41.750+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:42.037 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:41.750+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:42.037 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:41.750+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:42.038 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:41.751+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:42.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:42.036+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:42.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:42.036+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:42.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:42.036+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:42.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:42.037+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:42.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:42.398+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:42.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:42.398+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:42.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:42.398+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:42.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:42.399+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:43.068 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:42.723+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:43.069 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:42.723+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:43.069 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:42.723+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:43.069 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:42.724+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:43.068+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:43.068+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:43.068+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:43.069+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:43.770 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:43.444+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:43.771 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:43.444+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:43.771 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:43.444+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:43.771 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:43.445+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:44.077 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:43.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:44.078 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:43.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:44.078 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:43.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:44.078 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:43.769+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:44.077+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:44.077+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:44.077+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:44.078+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:44.815 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:44.411+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:44.815 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:44.411+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:44.815 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:44.411+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:44.815 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:44.412+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:45.073 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:44.814+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:45.073 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:44.814+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:45.073 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:44.814+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:45.073 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:44.815+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:45.072+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:45.072+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:45.072+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:45.073+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:45.780 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:45.423+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:45.780 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:45.423+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:45.781 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:45.423+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:45.781 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:45.423+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:46.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:45.780+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:46.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:45.780+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:46.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:45.780+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:46.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:45.781+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:46.381 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:49:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:49:46.152+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 33130 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:49:46.381 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:46.122+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:46.382 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:46.122+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:46.382 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:46.122+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:46.382 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:46.123+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:46.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:46.381+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:46.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:46.381+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:46.784 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:46.381+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:46.784 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:46.382+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:47.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:46.782+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:47.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:46.782+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:47.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:46.782+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:47.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:46.783+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:47.407 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:47.091+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:47.407 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:47.091+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:47.407 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:47.091+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:47.407 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:47.092+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:47.747 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:47.406+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:47.748 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:47.406+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:47.748 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:47.406+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:47.748 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:47.407+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:48.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:47.747+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:48.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:47.747+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:48.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:47.747+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:48.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:47.748+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:48.421 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:48.129+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:48.421 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:48.129+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:48.421 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:48.129+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:48.421 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:48.130+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:48.751 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:48.420+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:48.751 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:48.420+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:48.751 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:48.420+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:48.751 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:48.421+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:48.750+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:48.750+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:48.750+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:48.751+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:49.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:49.134+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:49.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:49.134+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:49.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:49.134+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:49.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:49.134+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:49.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:49.450+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:49.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:49.450+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:49.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:49.450+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:49.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:49.451+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:49.720+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:49.720+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:49.720+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:50.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:49.721+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:50.425 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:50.124+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:50.426 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:50.124+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:50.426 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:50.124+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:50.426 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:50.125+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:50.753 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:50.425+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:50.753 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:50.425+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:50.753 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:50.425+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:50.753 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:50.426+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:51.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:50.752+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:51.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:50.752+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:51.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:50.752+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:51.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:50.753+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:51.469 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:49:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:49:51.156+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 33221 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:49:51.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:51.117+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:51.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:51.117+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:51.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:51.117+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:51.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:51.118+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:51.759 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:51.469+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:51.759 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:51.469+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:51.759 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:51.469+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:51.759 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:51.470+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:51.760 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:51.718+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:51.760 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:51.718+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:51.760 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:51.718+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:51.760 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:51.719+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:52.462 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:52.095+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:52.463 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:52.095+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:52.463 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:52.095+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:52.463 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:52.096+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:52.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:52.462+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:52.726 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:52.462+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:52.726 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:52.462+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:52.726 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:52.463+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:53.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:52.725+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:53.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:52.725+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:53.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:52.725+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:53.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:52.726+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:53.510 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:53.092+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:53.511 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:53.092+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:53.511 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:53.092+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:53.511 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:53.093+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:53.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:53.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:53.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:53.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:53.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:53.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:53.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:53.511+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:53.696+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:53.696+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:53.696+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:53.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:53.697+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:54.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:54.070+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:54.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:54.070+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:54.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:54.070+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:54.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:54.070+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:54.552+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:54.552+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:54.552+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:54.553+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:54.737+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:54.737+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:54.737+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:54.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:54.737+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:55.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:55.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:55.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:55.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:55.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:55.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:55.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:55.033+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:55.516+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:55.516+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:55.516+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:55.516+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:55.710+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:55.710+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:55.710+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:55.711+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:56.161 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:56.017+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:56.162 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:56.017+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:56.162 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:56.017+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:56.162 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:56.018+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:56.281 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-19T03:49:56.282 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-03-19T03:49:56.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:49:56.283 DEBUG:teuthology.orchestra.run.smithi059:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-03-19T03:49:56.331 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/behavior: Cannot mkdir: Input/output error 2024-03-19T03:49:56.331 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/behavior/WIFEXITED: Cannot mkdir: Input/output error 2024-03-19T03:49:56.331 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/behavior/WIFEXITED/1-2.c: Cannot open: Input/output error 2024-03-19T03:49:56.331 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/behavior/WIFEXITED/1-1.c: Cannot open: Input/output error 2024-03-19T03:49:56.332 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/behavior/WIFEXITED/assertions.xml: Cannot open: Input/output error 2024-03-19T03:49:56.332 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/behavior/WIFEXITED/1-3.c: Cannot open: Input/output error 2024-03-19T03:49:56.332 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/behavior/timers: Cannot mkdir: Input/output error 2024-03-19T03:49:56.332 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/behavior/timers/coverage.txt: Cannot open: Input/output error 2024-03-19T03:49:56.332 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/behavior/timers/2-1.c: Cannot open: Input/output error 2024-03-19T03:49:56.332 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/behavior/timers/1-1.c: Cannot open: Input/output error 2024-03-19T03:49:56.332 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/behavior/timers/assertions.xml: Cannot open: Input/output error 2024-03-19T03:49:56.332 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance: Cannot utime: Input/output error 2024-03-19T03:49:56.333 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.333 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/execute.sh: Cannot open: Input/output error 2024-03-19T03:49:56.333 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional: Cannot mkdir: Input/output error 2024-03-19T03:49:56.333 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/mqueues: Cannot mkdir: Input/output error 2024-03-19T03:49:56.333 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/mqueues/send_rev_2.c: Cannot open: Input/output error 2024-03-19T03:49:56.333 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/mqueues/coverage.txt: Cannot open: Input/output error 2024-03-19T03:49:56.333 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/mqueues/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.333 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/mqueues/send_rev_1.c: Cannot open: Input/output error 2024-03-19T03:49:56.333 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/mqueues/assertions.xml: Cannot open: Input/output error 2024-03-19T03:49:56.334 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/mqueues/run.sh: Cannot open: Input/output error 2024-03-19T03:49:56.334 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/mqueues/plan.txt: Cannot open: Input/output error 2024-03-19T03:49:56.334 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/semaphores: Cannot mkdir: Input/output error 2024-03-19T03:49:56.334 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/semaphores/sem_conpro.c: Cannot open: Input/output error 2024-03-19T03:49:56.334 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/semaphores/sem_lock.c: Cannot open: Input/output error 2024-03-19T03:49:56.334 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/semaphores/coverage.txt: Cannot open: Input/output error 2024-03-19T03:49:56.334 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/semaphores/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.334 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/semaphores/sem_sleepingbarber.c: Cannot open: Input/output error 2024-03-19T03:49:56.335 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/semaphores/assertions.xml: Cannot open: Input/output error 2024-03-19T03:49:56.335 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/semaphores/sem_readerwriter.c: Cannot open: Input/output error 2024-03-19T03:49:56.335 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/semaphores/sem_philosopher.c: Cannot open: Input/output error 2024-03-19T03:49:56.335 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/semaphores/run.sh: Cannot open: Input/output error 2024-03-19T03:49:56.335 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/semaphores/plan.txt: Cannot open: Input/output error 2024-03-19T03:49:56.335 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/timers: Cannot mkdir: Input/output error 2024-03-19T03:49:56.335 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/timers/coverage.txt: Cannot open: Input/output error 2024-03-19T03:49:56.335 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/timers/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.335 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/timers/assertions.xml: Cannot open: Input/output error 2024-03-19T03:49:56.336 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/timers/clocks: Cannot mkdir: Input/output error 2024-03-19T03:49:56.336 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/timers/clocks/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.336 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/timers/clocks/twopsetclock.c: Cannot open: Input/output error 2024-03-19T03:49:56.336 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/timers/clocks/invaliddates.c: Cannot open: Input/output error 2024-03-19T03:49:56.336 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/timers/timers: Cannot mkdir: Input/output error 2024-03-19T03:49:56.336 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/timers/timers/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.336 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/timers/timers/twoevtimers.c: Cannot open: Input/output error 2024-03-19T03:49:56.336 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/timers/timers/twoptimers.c: Cannot open: Input/output error 2024-03-19T03:49:56.336 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/timers/run.sh: Cannot open: Input/output error 2024-03-19T03:49:56.337 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/timers/plan.txt: Cannot open: Input/output error 2024-03-19T03:49:56.337 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads: Cannot mkdir: Input/output error 2024-03-19T03:49:56.337 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/condvar: Cannot mkdir: Input/output error 2024-03-19T03:49:56.337 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/condvar/pthread_cond_wait_2.c: Cannot open: Input/output error 2024-03-19T03:49:56.337 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/condvar/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.337 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/condvar/pthread_cond_wait_1.c: Cannot open: Input/output error 2024-03-19T03:49:56.337 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/robust_test: Cannot mkdir: Input/output error 2024-03-19T03:49:56.337 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/robust_test/robust2-sun.c: Cannot open: Input/output error 2024-03-19T03:49:56.337 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/robust_test/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.338 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/robust_test/robust3-mode2.c: Cannot open: Input/output error 2024-03-19T03:49:56.338 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/robust_test/README: Cannot open: Input/output error 2024-03-19T03:49:56.338 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/robust_test/robust1-sun.c: Cannot open: Input/output error 2024-03-19T03:49:56.338 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/robust_test/robust1-mode2.c: Cannot open: Input/output error 2024-03-19T03:49:56.338 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/robust_test/run.sh: Cannot open: Input/output error 2024-03-19T03:49:56.338 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/robust_test/robust2-mode2.c: Cannot open: Input/output error 2024-03-19T03:49:56.338 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/schedule: Cannot mkdir: Input/output error 2024-03-19T03:49:56.338 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/schedule/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.339 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/schedule/1-2.c: Cannot open: Input/output error 2024-03-19T03:49:56.339 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/schedule/1-1.c: Cannot open: Input/output error 2024-03-19T03:49:56.339 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.339 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/tools: Cannot mkdir: Input/output error 2024-03-19T03:49:56.339 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/tools/watchdogtimer.sh: Cannot open: Input/output error 2024-03-19T03:49:56.339 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/README: Cannot open: Input/output error 2024-03-19T03:49:56.339 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/pi_test: Cannot mkdir: Input/output error 2024-03-19T03:49:56.340 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/pi_test/pitest-6.c: Cannot open: Input/output error 2024-03-19T03:49:56.340 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/pi_test/pitest-5.c: Cannot open: Input/output error 2024-03-19T03:49:56.340 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/pi_test/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.340 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/pi_test/pitest-2.c: Cannot open: Input/output error 2024-03-19T03:49:56.341 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/pi_test/pitest-4.c: Cannot open: Input/output error 2024-03-19T03:49:56.341 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/pi_test/README: Cannot open: Input/output error 2024-03-19T03:49:56.341 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/pi_test/pitest-1.c: Cannot open: Input/output error 2024-03-19T03:49:56.341 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/pi_test/do-plot: Cannot open: Input/output error 2024-03-19T03:49:56.342 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/pi_test/pitest-3.c: Cannot open: Input/output error 2024-03-19T03:49:56.342 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/pi_test/run.sh: Cannot open: Input/output error 2024-03-19T03:49:56.342 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/run.sh: Cannot open: Input/output error 2024-03-19T03:49:56.342 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/include: Cannot mkdir: Input/output error 2024-03-19T03:49:56.342 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/include/test.h: Cannot open: Input/output error 2024-03-19T03:49:56.342 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/include/pitest.h: Cannot open: Input/output error 2024-03-19T03:49:56.343 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/functional/threads/Makefile.inc: Cannot open: Input/output error 2024-03-19T03:49:56.343 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/README: Cannot open: Input/output error 2024-03-19T03:49:56.343 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/exec-func.sh: Cannot open: Input/output error 2024-03-19T03:49:56.343 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/Documentation: Cannot mkdir: Input/output error 2024-03-19T03:49:56.343 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/Documentation/COVERAGE.threads: Cannot open: Input/output error 2024-03-19T03:49:56.343 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/Documentation/COVERAGE.signals: Cannot open: Input/output error 2024-03-19T03:49:56.343 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/Documentation/HOWTO_RunTests: Cannot open: Input/output error 2024-03-19T03:49:56.343 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/Documentation/HOWTO_Release: Cannot open: Input/output error 2024-03-19T03:49:56.343 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/Documentation/HOWTO_FunctionalStressTest: Cannot open: Input/output error 2024-03-19T03:49:56.344 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/Documentation/COVERAGE.mqueues: Cannot open: Input/output error 2024-03-19T03:49:56.344 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/Documentation/HOWTO_ConformanceTest: Cannot open: Input/output error 2024-03-19T03:49:56.344 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/Documentation/HOWTO_INDEX: Cannot open: Input/output error 2024-03-19T03:49:56.344 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/Documentation/HOWTO_Tagging: Cannot open: Input/output error 2024-03-19T03:49:56.344 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/Documentation/COVERAGE.timers: Cannot open: Input/output error 2024-03-19T03:49:56.344 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/Documentation/HOWTO_Coverage: Cannot open: Input/output error 2024-03-19T03:49:56.344 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/Documentation/COVERAGE.semaphores: Cannot open: Input/output error 2024-03-19T03:49:56.344 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/Documentation/HOWTO_Assertions: Cannot open: Input/output error 2024-03-19T03:49:56.344 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/Documentation/HOWTO_CodingGuidelines: Cannot open: Input/output error 2024-03-19T03:49:56.345 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/Documentation/HOWTO_BugReporting: Cannot open: Input/output error 2024-03-19T03:49:56.345 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/Documentation/HOWTO_DefinitionsTest: Cannot open: Input/output error 2024-03-19T03:49:56.345 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/Documentation/HOWTO_BoundaryTest: Cannot open: Input/output error 2024-03-19T03:49:56.345 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/Documentation/HOWTO_ResultCodes: Cannot open: Input/output error 2024-03-19T03:49:56.345 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/QUICK-START: Cannot open: Input/output error 2024-03-19T03:49:56.345 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/NEWS: Cannot open: Input/output error 2024-03-19T03:49:56.345 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/locate-test: Cannot open: Input/output error 2024-03-19T03:49:56.345 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/LDFLAGS: Cannot open: Input/output error 2024-03-19T03:49:56.346 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress: Cannot mkdir: Input/output error 2024-03-19T03:49:56.346 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/signals: Cannot mkdir: Input/output error 2024-03-19T03:49:56.346 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/signals/sigismember_stress_1.c: Cannot open: Input/output error 2024-03-19T03:49:56.346 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/mqueues: Cannot mkdir: Input/output error 2024-03-19T03:49:56.346 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/mqueues/multi_send_rev_1.c: Cannot open: Input/output error 2024-03-19T03:49:56.346 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/mqueues/coverage.txt: Cannot open: Input/output error 2024-03-19T03:49:56.346 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/mqueues/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.346 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/mqueues/assertions.xml: Cannot open: Input/output error 2024-03-19T03:49:56.346 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/mqueues/multi_send_rev_2.c: Cannot open: Input/output error 2024-03-19T03:49:56.347 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/mqueues/run.sh: Cannot open: Input/output error 2024-03-19T03:49:56.347 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/mqueues/plan.txt: Cannot open: Input/output error 2024-03-19T03:49:56.347 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/semaphores: Cannot mkdir: Input/output error 2024-03-19T03:49:56.347 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/semaphores/coverage.txt: Cannot open: Input/output error 2024-03-19T03:49:56.347 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/semaphores/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.347 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/semaphores/assertions.xml: Cannot open: Input/output error 2024-03-19T03:49:56.347 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/semaphores/multi_con_pro.c: Cannot open: Input/output error 2024-03-19T03:49:56.348 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/semaphores/run.sh: Cannot open: Input/output error 2024-03-19T03:49:56.348 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/semaphores/plan.txt: Cannot open: Input/output error 2024-03-19T03:49:56.348 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/timers: Cannot mkdir: Input/output error 2024-03-19T03:49:56.348 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/timers/keytests.sh: Cannot open: Input/output error 2024-03-19T03:49:56.348 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/timers/plan.txt: Cannot open: Input/output error 2024-03-19T03:49:56.348 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads: Cannot mkdir: Input/output error 2024-03-19T03:49:56.348 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_init: Cannot mkdir: Input/output error 2024-03-19T03:49:56.348 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_init/s-c1.c: Cannot open: Input/output error 2024-03-19T03:49:56.348 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_init/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.349 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_init/testfrmw.c: Cannot open: Input/output error 2024-03-19T03:49:56.349 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_init/testfrmw.h: Cannot open: Input/output error 2024-03-19T03:49:56.349 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_init/do-plot: Cannot open: Input/output error 2024-03-19T03:49:56.349 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_init/NOTES: Cannot open: Input/output error 2024-03-19T03:49:56.349 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_init/LICENCE: Cannot open: Input/output error 2024-03-19T03:49:56.349 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_getschedparam: Cannot mkdir: Input/output error 2024-03-19T03:49:56.349 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_getschedparam/stress.c: Cannot open: Input/output error 2024-03-19T03:49:56.349 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_getschedparam/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.350 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_getschedparam/testfrmw.c: Cannot open: Input/output error 2024-03-19T03:49:56.350 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_getschedparam/testfrmw.h: Cannot open: Input/output error 2024-03-19T03:49:56.350 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_getschedparam/NOTES: Cannot open: Input/output error 2024-03-19T03:49:56.350 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_getschedparam/LICENCE: Cannot open: Input/output error 2024-03-19T03:49:56.350 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_open: Cannot mkdir: Input/output error 2024-03-19T03:49:56.350 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_open/s-c1.c: Cannot open: Input/output error 2024-03-19T03:49:56.350 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_open/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.350 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_open/testfrmw.c: Cannot open: Input/output error 2024-03-19T03:49:56.351 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_open/testfrmw.h: Cannot open: Input/output error 2024-03-19T03:49:56.351 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_open/do-plot: Cannot open: Input/output error 2024-03-19T03:49:56.351 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_open/NOTES: Cannot open: Input/output error 2024-03-19T03:49:56.351 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_open/LICENCE: Cannot open: Input/output error 2024-03-19T03:49:56.351 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_getvalue: Cannot mkdir: Input/output error 2024-03-19T03:49:56.351 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_getvalue/stress.c: Cannot open: Input/output error 2024-03-19T03:49:56.351 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_getvalue/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.351 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_getvalue/testfrmw.c: Cannot open: Input/output error 2024-03-19T03:49:56.351 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_getvalue/testfrmw.h: Cannot open: Input/output error 2024-03-19T03:49:56.352 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_getvalue/NOTES: Cannot open: Input/output error 2024-03-19T03:49:56.352 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/sem_getvalue/LICENCE: Cannot open: Input/output error 2024-03-19T03:49:56.352 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_exit: Cannot mkdir: Input/output error 2024-03-19T03:49:56.352 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_exit/stress.c: Cannot open: Input/output error 2024-03-19T03:49:56.352 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_exit/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.352 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_exit/testfrmw.c: Cannot open: Input/output error 2024-03-19T03:49:56.352 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_exit/testfrmw.h: Cannot open: Input/output error 2024-03-19T03:49:56.352 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_exit/NOTES: Cannot open: Input/output error 2024-03-19T03:49:56.352 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_exit/LICENCE: Cannot open: Input/output error 2024-03-19T03:49:56.353 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_exit/threads_scenarii.c: Cannot open: Input/output error 2024-03-19T03:49:56.353 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_trylock: Cannot mkdir: Input/output error 2024-03-19T03:49:56.353 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_trylock/stress.c: Cannot open: Input/output error 2024-03-19T03:49:56.353 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_trylock/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.353 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_trylock/testfrmw.c: Cannot open: Input/output error 2024-03-19T03:49:56.353 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_trylock/testfrmw.h: Cannot open: Input/output error 2024-03-19T03:49:56.353 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_trylock/NOTES: Cannot open: Input/output error 2024-03-19T03:49:56.353 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_trylock/LICENCE: Cannot open: Input/output error 2024-03-19T03:49:56.353 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_init: Cannot mkdir: Input/output error 2024-03-19T03:49:56.354 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_init/stress.c: Cannot open: Input/output error 2024-03-19T03:49:56.354 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_init/s-c.c: Cannot open: Input/output error 2024-03-19T03:49:56.354 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_init/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.354 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_init/testfrmw.c: Cannot open: Input/output error 2024-03-19T03:49:56.354 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_init/testfrmw.h: Cannot open: Input/output error 2024-03-19T03:49:56.354 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_init/NOTES: Cannot open: Input/output error 2024-03-19T03:49:56.355 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_init/LICENCE: Cannot open: Input/output error 2024-03-19T03:49:56.355 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_lock: Cannot mkdir: Input/output error 2024-03-19T03:49:56.355 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_lock/stress.c: Cannot open: Input/output error 2024-03-19T03:49:56.355 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_lock/s-c1.c: Cannot open: Input/output error 2024-03-19T03:49:56.355 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_lock/s-c2.c: Cannot open: Input/output error 2024-03-19T03:49:56.355 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_lock/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.355 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_lock/testfrmw.c: Cannot open: Input/output error 2024-03-19T03:49:56.355 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_lock/testfrmw.h: Cannot open: Input/output error 2024-03-19T03:49:56.356 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_lock/NOTES: Cannot open: Input/output error 2024-03-19T03:49:56.356 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_mutex_lock/LICENCE: Cannot open: Input/output error 2024-03-19T03:49:56.356 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/fork: Cannot mkdir: Input/output error 2024-03-19T03:49:56.356 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/fork/s-c1.c: Cannot open: Input/output error 2024-03-19T03:49:56.356 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/fork/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.356 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/fork/testfrmw.c: Cannot open: Input/output error 2024-03-19T03:49:56.356 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/fork/testfrmw.h: Cannot open: Input/output error 2024-03-19T03:49:56.356 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/fork/do-plot: Cannot open: Input/output error 2024-03-19T03:49:56.356 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/fork/NOTES: Cannot open: Input/output error 2024-03-19T03:49:56.357 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/fork/LICENCE: Cannot open: Input/output error 2024-03-19T03:49:56.357 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_cond_wait: Cannot mkdir: Input/output error 2024-03-19T03:49:56.357 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_cond_wait/stress.c: Cannot open: Input/output error 2024-03-19T03:49:56.357 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_cond_wait/stress1.c: Cannot open: Input/output error 2024-03-19T03:49:56.357 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_cond_wait/testfrmw.c: Cannot open: Input/output error 2024-03-19T03:49:56.357 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_cond_wait/testfrmw.h: Cannot open: Input/output error 2024-03-19T03:49:56.357 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_cond_wait/stress2.c: Cannot open: Input/output error 2024-03-19T03:49:56.357 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_kill: Cannot mkdir: Input/output error 2024-03-19T03:49:56.358 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_kill/stress.c: Cannot open: Input/output error 2024-03-19T03:49:56.358 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_kill/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.358 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_kill/testfrmw.c: Cannot open: Input/output error 2024-03-19T03:49:56.358 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_kill/testfrmw.h: Cannot open: Input/output error 2024-03-19T03:49:56.358 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_kill/NOTES: Cannot open: Input/output error 2024-03-19T03:49:56.358 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_kill/LICENCE: Cannot open: Input/output error 2024-03-19T03:49:56.358 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_cancel: Cannot mkdir: Input/output error 2024-03-19T03:49:56.358 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_cancel/stress.c: Cannot open: Input/output error 2024-03-19T03:49:56.359 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_cancel/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.359 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_cancel/testfrmw.c: Cannot open: Input/output error 2024-03-19T03:49:56.359 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_cancel/testfrmw.h: Cannot open: Input/output error 2024-03-19T03:49:56.359 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_cancel/NOTES: Cannot open: Input/output error 2024-03-19T03:49:56.359 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_cancel/LICENCE: Cannot open: Input/output error 2024-03-19T03:49:56.359 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/assertions.xml: Cannot open: Input/output error 2024-03-19T03:49:56.359 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_cond_init: Cannot mkdir: Input/output error 2024-03-19T03:49:56.359 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_cond_init/stress.c: Cannot open: Input/output error 2024-03-19T03:49:56.359 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_cond_init/s-c.c: Cannot open 2024-03-19T03:49:56.360 INFO:tasks.workunit.client.0.smithi059.stderr:gzip: : Input/output error 2024-03-19T03:49:56.360 INFO:tasks.workunit.client.0.smithi059.stderr:stdin: Stale file handle 2024-03-19T03:49:56.360 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_cond_init/Makefile: Cannot open: Input/output error 2024-03-19T03:49:56.360 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/stress/threads/pthread_cond_init/testfrmw.c: Cannot open: Input/output error 2024-03-19T03:49:56.360 INFO:tasks.workunit.client.0.smithi059.stderr:tar: Unexpected EOF in archive 2024-03-19T03:49:56.360 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite: Cannot utime: Input/output error 2024-03-19T03:49:56.360 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases: Cannot utime: Input/output error 2024-03-19T03:49:56.361 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231: Cannot utime: Input/output error 2024-03-19T03:49:56.361 INFO:tasks.workunit.client.0.smithi059.stderr:tar: Error is not recoverable: exiting now 2024-03-19T03:49:56.362 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-19T03:49:56.362 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-03-19T03:49:56.362 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-19T03:49:56.363 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-19T03:49:56.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:49:56.365 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-19T03:49:56.428 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:49:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:49:56.161+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 33312 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:49:56.437 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-19T03:49:56.438 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T03:49:56.688 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:56.491+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:56.689 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:56.491+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:56.689 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:56.491+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:56.689 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:56.492+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:57.009 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:56.709+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:57.009 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:56.709+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:57.009 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:56.709+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:57.010 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:56.710+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:57.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:57.008+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:57.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:57.008+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:57.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:57.008+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:57.009+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:57.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:57.453+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:57.453+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:57.453+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:57.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:57.454+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:57.686+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:57.686+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:57.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:57.686+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:57.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:57.687+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:57.999+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:58.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:58.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:58.000+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:58.425+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:58.425+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:58.425+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:58.425+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:58.661+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:58.661+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:58.661+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:58.661+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:49:59.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:58.963+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:59.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:58.963+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:59.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:58.963+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:49:59.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:58.964+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:59.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:59.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:59.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:49:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:49:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:49:59.450+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:49:59.942 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:59.678+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:59.942 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:59.678+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:59.942 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:59.678+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:49:59.942 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:49:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:49:59.679+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:00.219 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:59.941+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:00.219 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:59.941+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:00.220 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:59.941+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:00.220 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:49:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:49:59.942+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:00.428+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:00.428+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:00.428+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:00.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:00.429+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:01.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:00.909+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:01.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:00.909+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:01.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:00.910+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:00.910+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:00.724+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:00.724+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:00.724+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:01.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:00.724+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:01.452 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:50:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:50:01.165+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 33402 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:50:01.735 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:01.451+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:01.736 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:01.451+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:01.736 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:01.451+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:01.736 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:01.452+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:02.020 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:01.735+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:02.020 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:01.735+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:02.020 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:01.735+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:02.020 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:01.736+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:02.020 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:01.874+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:02.021 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:01.874+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:02.021 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:01.874+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:02.021 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:01.875+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:02.739 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:02.425+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:02.739 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:02.425+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:02.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:02.425+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:02.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:02.426+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:03.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:02.874+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:03.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:02.874+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:03.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:02.874+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:02.875+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:02.739+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:02.739+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:02.739+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:02.740+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:03.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:03.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:03.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:03.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:03.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:03.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:03.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:03.468+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:04.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:03.784+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:04.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:03.784+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:04.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:03.784+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:04.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:03.786+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:04.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:03.856+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:04.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:03.856+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:04.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:03.856+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:04.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:03.857+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:04.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:04.431+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:04.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:04.431+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:04.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:04.431+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:04.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:04.432+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:04.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:04.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:04.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:04.795+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:04.842+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:04.842+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:04.842+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:04.843+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:05.761 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:05.419+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:05.761 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:05.419+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:05.761 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:05.419+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:05.761 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:05.420+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:06.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:05.760+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:06.091 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:05.760+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:06.091 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:05.760+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:06.091 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:05.762+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:06.092 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:05.814+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:06.092 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:05.814+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:06.092 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:05.814+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:06.093 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:05.815+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:06.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:50:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:50:06.169+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 33493 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:50:06.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:06.392+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:06.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:06.392+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:06.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:06.392+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:06.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:06.392+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:07.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:06.727+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:07.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:06.727+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:07.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:06.727+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:07.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:06.729+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:06.779+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:06.779+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:06.779+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:06.780+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:07.739 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:07.379+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:07.739 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:07.379+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:07.739 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:07.379+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:07.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:07.379+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:08.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:07.738+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:08.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:07.738+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:08.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:07.738+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:08.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:07.739+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:07.761+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:07.761+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:07.761+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:07.762+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:08.714 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:08.389+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:08.715 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:08.389+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:08.715 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:08.389+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:08.715 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:08.390+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:09.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:08.757+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:09.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:08.757+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:09.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:08.757+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:09.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:08.758+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:08.714+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:08.714+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:08.714+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:08.715+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:09.349+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:09.349+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:09.349+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:09.350+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:10.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:09.688+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:10.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:09.688+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:10.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:09.688+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:10.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:09.689+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:09.793+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:09.793+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:09.793+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:09.794+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:10.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:10.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:10.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:10.340+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:10.700+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:10.700+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:10.700+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:10.701+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:10.767+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:10.767+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:10.767+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:10.768+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:11.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:50:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:50:11.173+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 33584 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:50:11.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:11.308+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:11.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:11.308+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:11.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:11.308+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:11.309+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:12.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:11.715+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:12.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:11.715+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:12.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:11.715+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:12.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:11.715+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:12.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:11.732+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:12.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:11.732+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:12.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:11.732+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:12.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:11.733+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:12.731 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:12.353+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:12.731 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:12.353+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:12.732 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:12.353+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:12.732 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:12.354+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:12.731+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:12.731+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:12.731+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:12.731+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:12.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:12.763+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:12.763+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:12.763+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:13.720 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:13.386+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:13.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:13.386+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:13.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:13.386+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:13.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:13.388+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:13.770+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:13.770+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:13.770+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:14.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:13.771+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:13.720+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:13.720+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:13.720+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:13.721+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:14.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:14.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:14.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:14.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:14.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:14.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:14.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:14.396+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:15.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:14.782+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:15.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:14.782+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:15.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:14.782+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:15.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:14.783+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:14.748+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:14.748+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:14.748+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:14.748+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:15.706 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:15.425+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:15.706 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:15.425+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:15.707 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:15.425+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:15.707 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:15.426+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:16.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:15.704+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:16.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:15.704+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:16.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:15.704+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:16.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:15.705+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:15.798+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:15.798+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:15.798+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:15.799+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:16.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:50:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:50:16.177+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 33675 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:50:16.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:16.402+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:16.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:16.402+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:16.402+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:16.403+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:17.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:16.809+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:17.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:16.809+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:17.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:16.809+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:17.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:16.810+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:17.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:16.702+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:17.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:16.702+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:17.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:16.702+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:17.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:16.702+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:17.802 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:17.427+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:17.803 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:17.427+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:17.803 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:17.427+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:17.803 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:17.427+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:17.804 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:17.653+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:17.804 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:17.653+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:17.804 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:17.653+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:17.805 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:17.654+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:18.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:17.802+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:18.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:17.802+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:18.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:17.802+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:18.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:17.803+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:18.792 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:18.434+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:18.792 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:18.434+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:18.792 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:18.434+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:18.793 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:18.435+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:18.793 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:18.615+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:18.793 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:18.615+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:18.793 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:18.615+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:18.793 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:18.615+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:19.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:18.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:19.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:18.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:19.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:18.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:19.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:18.792+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:19.443+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:19.443+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:19.443+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:19.444+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:20.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:19.652+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:20.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:19.652+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:20.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:19.652+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:20.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:19.653+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:20.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:19.777+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:20.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:19.777+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:20.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:19.777+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:20.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:19.778+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:20.796 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:20.616+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:20.796 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:20.616+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:20.796 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:20.616+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:20.797 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:20.617+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:20.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:20.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:20.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:20.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:20.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:20.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:20.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:20.479+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:21.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:20.796+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:21.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:20.796+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:21.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:20.796+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:21.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:20.796+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:21.502 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:50:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:50:21.182+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 33765 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:50:21.760 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:21.609+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:21.760 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:21.609+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:21.761 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:21.609+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:21.761 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:21.610+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:21.761 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:21.501+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:21.761 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:21.501+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:21.761 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:21.501+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:21.761 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:21.502+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:21.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:21.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:21.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:21.792+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:22.775 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:22.490+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:22.775 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:22.490+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:22.775 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:22.490+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:22.775 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:22.491+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:22.775 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:22.621+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:22.776 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:22.621+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:22.776 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:22.621+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:22.776 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:22.622+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:22.774+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:22.774+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:22.774+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:22.775+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:23.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:23.619+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:23.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:23.619+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:23.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:23.619+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:23.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:23.620+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:23.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:23.484+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:23.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:23.484+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:23.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:23.484+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:23.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:23.485+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:23.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:23.732+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:23.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:23.732+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:23.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:23.732+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:23.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:23.733+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:24.751 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:24.481+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:24.752 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:24.481+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:24.752 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:24.481+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:24.752 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:24.482+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:24.752 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:24.614+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:24.752 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:24.614+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:24.752 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:24.614+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:24.753 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:24.615+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:25.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:24.751+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:25.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:24.751+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:25.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:24.751+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:25.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:24.752+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:25.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:25.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:25.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:25.506+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:25.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:25.570+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:25.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:25.570+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:25.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:25.570+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:25.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:25.571+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:25.712+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:25.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:25.712+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:25.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:25.712+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:25.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:25.713+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:26.460 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:50:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:50:26.186+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 33856 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:50:26.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:26.605+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:26.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:26.605+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:26.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:26.605+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:26.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:26.606+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:26.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:26.460+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:26.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:26.460+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:26.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:26.460+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:26.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:26.460+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:26.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:26.672+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:26.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:26.672+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:26.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:26.672+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:26.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:26.673+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:27.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:27.504+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:27.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:27.504+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:27.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:27.504+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:27.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:27.505+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:27.595+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:27.595+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:27.595+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:27.596+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:27.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:27.677+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:27.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:27.677+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:27.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:27.677+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:27.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:27.678+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:28.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:28.716+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:28.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:28.716+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:28.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:28.716+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:28.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:28.716+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:28.498+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:28.498+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:28.498+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:28.498+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:28.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:28.558+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:28.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:28.558+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:28.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:28.558+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:28.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:28.558+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:29.453+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:29.453+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:29.453+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:29.454+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:30.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:29.729+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:30.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:29.729+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:30.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:29.729+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:30.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:29.730+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:30.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:29.606+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:30.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:29.606+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:30.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:29.606+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:30.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:29.606+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:30.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:30.419+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:30.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:30.419+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:30.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:30.419+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:30.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:30.420+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:31.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:30.734+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:31.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:30.734+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:31.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:30.734+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:31.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:30.735+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:30.637+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:30.637+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:30.637+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:30.638+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:31.468 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:50:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:50:31.191+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 33947 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:50:31.761 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:31.469+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:31.761 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:31.469+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:31.762 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:31.469+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:31.762 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:31.470+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:31.762 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:31.608+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:31.762 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:31.608+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:31.762 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:31.608+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:31.763 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:31.609+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:31.763 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:31.717+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:31.763 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:31.717+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:31.763 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:31.717+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:31.763 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:31.717+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:32.739 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:32.483+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:32.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:32.483+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:32.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:32.483+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:32.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:32.484+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:32.740 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:32.623+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:32.740 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:32.623+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:32.741 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:32.623+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:32.741 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:32.623+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:33.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:32.746+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:33.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:32.746+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:33.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:32.746+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:33.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:32.747+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:33.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:33.636+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:33.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:33.636+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:33.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:33.636+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:33.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:33.637+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:33.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:33.450+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:33.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:33.450+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:33.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:33.450+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:33.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:33.451+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:33.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:33.698+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:33.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:33.698+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:33.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:33.698+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:33.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:33.698+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:34.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:34.689+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:34.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:34.689+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:34.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:34.689+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:34.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:34.690+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:34.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:34.497+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:34.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:34.497+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:34.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:34.497+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:34.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:34.497+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:34.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:34.681+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:34.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:34.681+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:34.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:34.681+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:34.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:34.681+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:35.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:35.703+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:35.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:35.703+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:35.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:35.703+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:35.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:35.704+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:35.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:35.544+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:35.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:35.544+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:35.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:35.544+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:35.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:35.544+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:35.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:35.700+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:35.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:35.700+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:35.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:35.700+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:35.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:35.701+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:36.545 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:50:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:50:36.196+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 34039 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:50:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:36.545+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:36.545+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:36.545+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:36.546+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:36.686+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:36.686+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:36.686+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:36.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:36.686+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:36.742+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:36.742+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:36.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:36.742+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:36.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:36.743+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:37.576+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:37.576+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:37.576+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:37.576+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:37.701+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:37.701+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:37.701+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:37.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:37.702+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:37.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:37.750+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:37.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:37.750+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:37.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:37.750+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:37.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:37.751+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:39.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:38.613+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:39.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:38.613+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:39.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:38.613+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:39.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:38.614+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:38.666+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:38.666+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:38.666+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:39.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:38.667+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:38.781+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:38.781+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:39.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:38.781+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:39.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:38.781+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:40.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:39.785+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:40.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:39.785+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:40.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:39.785+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:40.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:39.786+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:40.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:39.629+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:40.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:39.629+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:40.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:39.629+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:40.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:39.630+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:40.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:39.642+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:40.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:39.642+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:40.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:39.642+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:40.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:39.644+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:41.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:40.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:41.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:40.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:41.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:40.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:41.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:40.610+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:41.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:40.605+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:41.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:40.605+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:41.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:40.605+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:41.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:40.606+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:41.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:40.744+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:41.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:40.744+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:41.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:40.744+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:41.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:40.745+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:41.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:50:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:50:41.199+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 34129 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:50:41.955 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:41.644+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:41.956 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:41.644+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:41.956 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:41.644+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:41.956 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:41.645+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:41.956 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:41.610+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:41.956 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:41.610+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:41.957 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:41.610+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:41.957 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:41.611+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:41.957 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:41.711+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:41.957 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:41.711+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:41.957 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:41.711+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:41.957 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:41.712+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:43.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:42.642+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:43.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:42.642+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:43.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:42.642+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:43.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:42.643+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:43.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:42.611+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:43.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:42.611+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:43.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:42.611+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:43.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:42.612+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:43.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:42.719+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:43.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:42.719+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:43.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:42.719+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:43.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:42.720+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:44.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:43.665+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:44.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:43.665+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:44.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:43.665+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:44.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:43.666+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:44.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:43.605+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:44.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:43.605+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:44.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:43.605+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:44.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:43.606+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:44.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:43.729+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:44.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:43.729+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:44.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:43.729+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:44.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:43.730+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:44.648+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:44.648+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:44.648+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:44.649+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:45.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:44.680+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:45.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:44.680+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:45.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:44.680+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:45.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:44.681+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:44.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:44.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:45.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:44.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:45.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:44.755+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:46.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:45.716+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:46.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:45.716+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:46.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:45.716+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:46.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:45.717+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:46.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:45.680+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:46.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:45.680+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:46.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:45.680+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:46.091 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:45.681+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:46.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:45.746+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:46.092 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:45.746+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:46.092 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:45.746+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:46.092 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:45.747+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:46.566 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:50:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:50:46.203+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 34221 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:50:46.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:46.717+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:46.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:46.717+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:46.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:46.717+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:46.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:46.718+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:46.708+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:46.708+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:46.708+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:46.709+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:46.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:46.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:46.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:46.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:46.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:46.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:46.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:46.755+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:48.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:47.685+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:48.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:47.685+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:48.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:47.685+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:48.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:47.686+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:48.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:47.759+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:48.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:47.759+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:48.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:47.759+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:48.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:47.759+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:48.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:47.738+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:48.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:47.738+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:48.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:47.738+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:48.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:47.738+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:48.731+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:48.731+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:48.731+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:48.732+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:49.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:48.735+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:49.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:48.735+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:49.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:48.735+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:49.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:48.736+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:49.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:48.743+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:49.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:48.743+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:49.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:48.743+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:49.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:48.744+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:49.739+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:49.739+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:49.739+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:49.740+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:49.687+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:49.687+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:49.687+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:49.688+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:49.786+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:49.786+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:50.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:49.786+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:50.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:49.786+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:50.737+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:50.737+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:50.737+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:50.738+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:51.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:50.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:51.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:50.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:51.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:50.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:51.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:50.745+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:50.829+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:51.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:50.829+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:51.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:50.829+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:51.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:50.829+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:51.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:50:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:50:51.207+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 34312 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:50:52.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:51.753+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:52.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:51.753+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:52.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:51.753+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:52.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:51.754+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:51.743+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:51.743+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:51.743+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:52.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:51.744+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:52.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:51.872+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:52.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:51.872+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:52.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:51.872+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:52.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:51.872+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:53.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:52.763+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:53.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:52.763+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:53.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:52.763+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:53.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:52.764+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:53.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:52.720+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:53.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:52.720+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:53.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:52.720+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:53.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:52.721+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:53.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:52.845+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:53.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:52.845+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:53.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:52.845+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:53.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:52.846+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:54.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:53.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:54.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:53.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:54.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:53.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:54.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:53.706+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:54.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:53.743+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:54.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:53.743+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:54.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:53.743+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:54.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:53.743+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:54.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:53.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:54.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:53.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:54.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:53.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:54.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:53.858+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:55.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:54.679+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:55.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:54.679+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:55.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:54.679+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:55.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:54.680+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:55.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:54.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:55.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:54.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:55.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:54.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:55.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:54.743+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:55.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:54.837+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:55.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:54.837+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:55.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:54.837+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:55.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:54.837+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:56.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:55.639+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:56.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:55.639+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:56.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:55.639+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:56.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:55.640+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:56.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:55.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:56.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:55.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:56.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:55.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:56.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:55.769+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:56.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:55.881+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:56.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:55.881+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:56.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:55.881+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:56.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:55.882+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:56.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:50:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:50:56.211+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 34404 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:50:56.929 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:56.640+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:56.929 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:56.640+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:56.929 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:56.640+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:56.929 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:56.641+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:56.929 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:56.809+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:56.930 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:56.809+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:56.930 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:56.809+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:56.930 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:56.810+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:57.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:56.928+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:57.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:56.928+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:57.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:56.928+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:57.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:56.929+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:57.896 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:57.796+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:57.896 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:57.796+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:57.896 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:57.796+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:57.896 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:57.797+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:57.897 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:57.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:57.897 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:57.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:57.897 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:57.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:57.897 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:57.610+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:57.940+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:57.940+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:57.940+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:57.940+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:58.898 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:58.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:58.898 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:58.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:58.898 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:58.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:58.899 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:58.837+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:50:58.899 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:58.647+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:58.899 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:58.647+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:58.899 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:58.647+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:58.899 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:58.648+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:58.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:58.896+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:58.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:58.896+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:58.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:58.897+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:50:58.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:58.897+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:59.892 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:59.629+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:59.892 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:59.629+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:59.893 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:59.629+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:50:59.893 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:50:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:50:59.630+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:50:59.893 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:59.825+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:59.893 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:59.825+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:59.893 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:59.825+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:50:59.893 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:50:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:50:59.826+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:00.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:59.892+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:00.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:59.892+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:00.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:59.892+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:00.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:50:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:50:59.892+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:01.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:00.882+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:01.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:00.882+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:01.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:00.882+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:01.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:00.883+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:00.820+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:00.820+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:00.820+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:01.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:00.821+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:01.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:00.676+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:01.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:00.676+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:01.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:00.676+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:01.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:00.677+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:01.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:51:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:51:01.216+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 34494 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:51:02.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:01.663+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:02.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:01.663+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:02.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:01.663+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:02.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:01.664+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:02.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:01.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:02.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:01.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:02.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:01.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:02.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:01.860+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:01.857+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:02.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:01.857+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:02.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:01.857+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:02.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:01.857+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:03.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:02.822+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:03.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:02.822+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:03.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:02.822+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:03.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:02.822+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:02.903+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:02.903+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:02.903+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:03.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:02.903+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:03.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:02.702+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:03.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:02.702+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:03.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:02.702+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:03.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:02.703+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:04.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:03.905+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:04.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:03.905+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:04.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:03.905+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:04.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:03.906+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:04.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:03.731+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:04.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:03.731+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:04.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:03.731+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:04.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:03.732+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:04.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:03.782+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:04.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:03.782+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:04.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:03.782+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:04.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:03.782+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:05.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:04.764+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:05.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:04.764+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:05.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:04.764+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:05.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:04.765+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:04.868+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:04.868+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:04.868+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:04.869+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:04.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:04.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:05.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:04.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:05.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:04.795+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:06.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:05.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:06.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:05.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:06.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:05.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:06.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:05.848+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:06.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:05.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:06.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:05.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:06.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:05.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:06.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:05.756+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:06.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:05.809+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:06.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:05.809+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:06.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:05.809+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:06.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:05.810+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:06.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:51:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:51:06.220+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 34586 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:51:07.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:06.816+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:07.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:06.816+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:06.816+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:06.816+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:06.746+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:06.746+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:07.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:06.746+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:07.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:06.747+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:07.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:06.815+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:07.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:06.815+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:07.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:06.815+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:07.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:06.815+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:08.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:07.846+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:08.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:07.846+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:08.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:07.846+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:08.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:07.847+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:07.767+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:07.767+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:07.767+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:07.767+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:08.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:07.815+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:08.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:07.815+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:08.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:07.815+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:08.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:07.816+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:09.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:08.876+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:09.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:08.876+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:09.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:08.876+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:09.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:08.877+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:09.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:08.860+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:09.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:08.860+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:09.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:08.860+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:09.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:08.861+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:08.780+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:08.780+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:09.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:08.780+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:09.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:08.780+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:10.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:09.835+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:10.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:09.835+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:10.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:09.835+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:10.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:09.835+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:10.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:09.879+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:10.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:09.879+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:10.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:09.879+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:10.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:09.880+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:09.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:10.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:09.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:10.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:09.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:10.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:09.763+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:10.834+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:10.834+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:10.834+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:11.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:10.834+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:10.730+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:10.730+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:10.730+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:10.731+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:11.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:10.845+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:11.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:10.845+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:11.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:10.845+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:11.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:10.845+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:11.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:51:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:51:11.224+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 34677 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:51:12.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:11.834+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:12.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:11.834+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:12.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:11.834+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:12.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:11.835+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:11.815+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:11.815+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:11.815+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:11.816+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:12.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:11.764+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:12.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:11.764+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:12.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:11.764+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:12.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:11.765+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:12.719+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:12.719+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:12.719+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:12.720+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:12.845+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:12.845+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:12.845+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:12.845+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:12.838+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:13.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:12.838+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:13.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:12.838+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:13.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:12.839+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:14.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:13.746+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:14.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:13.746+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:14.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:13.746+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:14.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:13.747+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:14.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:13.855+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:14.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:13.855+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:14.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:13.855+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:14.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:13.855+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:14.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:13.819+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:14.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:13.819+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:14.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:13.819+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:14.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:13.820+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:15.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:14.816+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:15.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:14.816+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:15.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:14.816+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:15.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:14.817+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:14.747+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:14.747+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:14.747+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:14.748+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:14.872+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:15.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:14.872+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:15.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:14.872+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:15.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:14.872+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:16.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:15.771+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:16.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:15.771+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:15.771+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:15.771+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:16.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:15.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:16.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:15.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:16.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:15.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:16.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:15.858+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:16.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:15.760+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:16.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:15.760+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:16.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:15.760+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:16.092 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:15.761+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:16.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:51:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:51:16.228+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 34769 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:51:17.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:16.785+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:17.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:16.785+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:17.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:16.785+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:17.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:16.786+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:17.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:16.906+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:17.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:16.906+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:17.091 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:16.906+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:17.091 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:16.907+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:17.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:16.803+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:17.092 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:16.803+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:17.092 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:16.803+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:17.092 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:16.804+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:18.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:17.873+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:18.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:17.873+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:18.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:17.873+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:18.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:17.873+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:18.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:17.798+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:18.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:17.798+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:18.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:17.798+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:18.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:17.798+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:18.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:17.817+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:18.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:17.817+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:18.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:17.817+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:18.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:17.819+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:19.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:18.779+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:19.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:18.779+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:19.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:18.779+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:19.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:18.780+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:18.854+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:18.854+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:18.854+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:18.855+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:19.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:18.808+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:19.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:18.808+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:19.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:18.808+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:19.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:18.809+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:20.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:19.849+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:20.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:19.849+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:20.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:19.849+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:20.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:19.849+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:20.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:19.820+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:20.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:19.820+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:20.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:19.820+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:20.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:19.821+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:20.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:19.790+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:20.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:19.790+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:20.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:19.790+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:20.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:19.790+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:21.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:20.812+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:21.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:20.812+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:21.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:20.812+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:21.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:20.813+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:21.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:20.810+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:21.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:20.810+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:21.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:20.810+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:21.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:20.810+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:21.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:20.827+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:21.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:20.827+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:21.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:20.827+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:21.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:20.828+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:21.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:51:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:51:21.232+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 34859 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:51:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:21.826+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:21.826+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:21.826+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:21.827+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:22.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:21.864+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:22.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:21.864+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:22.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:21.864+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:22.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:21.864+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:22.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:21.792+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:22.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:21.792+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:22.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:21.792+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:22.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:21.793+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:22.830+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:22.830+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:22.830+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:22.831+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:23.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:22.850+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:23.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:22.850+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:23.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:22.850+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:23.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:22.851+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:23.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:22.814+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:23.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:22.814+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:23.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:22.814+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:23.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:22.815+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:24.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:23.872+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:24.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:23.872+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:24.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:23.872+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:24.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:23.873+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:24.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:23.868+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:24.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:23.868+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:24.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:23.868+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:24.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:23.869+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:24.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:23.843+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:24.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:23.843+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:24.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:23.843+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:24.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:23.844+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:25.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:24.880+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:25.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:24.880+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:25.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:24.880+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:25.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:24.880+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:25.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:24.872+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:25.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:24.872+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:25.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:24.872+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:25.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:24.873+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:25.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:24.868+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:25.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:24.868+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:25.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:24.868+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:25.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:24.869+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:26.238 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:25.842+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:26.238 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:25.842+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:26.238 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:25.842+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:26.238 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:25.843+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:26.238 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:25.869+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:26.239 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:25.869+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:26.239 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:25.869+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:26.239 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:25.870+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:26.239 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:25.887+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:26.239 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:25.887+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:26.240 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:25.887+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:26.240 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:25.888+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:26.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:51:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:51:26.237+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 34951 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:51:27.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:26.829+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:27.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:26.829+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:27.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:26.829+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:27.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:26.830+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:27.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:26.795+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:27.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:26.795+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:27.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:26.795+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:27.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:26.796+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:27.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:26.904+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:27.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:26.904+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:27.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:26.904+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:27.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:26.904+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:27.838+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:27.838+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:27.838+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:27.839+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:27.946+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:27.946+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:27.946+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:27.947+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:28.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:27.816+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:28.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:27.816+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:28.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:27.816+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:28.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:27.817+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:29.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:28.829+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:29.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:28.829+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:29.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:28.829+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:29.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:28.830+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:29.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:28.938+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:29.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:28.938+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:29.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:28.938+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:29.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:28.939+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:28.815+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:28.815+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:28.815+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:29.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:28.816+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:30.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:29.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:30.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:29.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:30.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:29.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:30.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:29.795+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:30.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:29.869+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:30.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:29.869+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:30.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:29.869+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:30.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:29.870+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:30.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:29.934+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:30.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:29.934+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:30.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:29.934+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:30.092 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:29.935+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:31.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:30.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:31.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:30.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:31.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:30.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:31.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:30.837+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:30.927+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:30.927+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:30.927+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:30.928+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:31.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:30.883+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:31.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:30.883+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:31.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:30.883+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:31.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:30.883+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:31.364 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:51:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:51:31.241+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 35042 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:51:32.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:31.863+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:32.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:31.863+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:32.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:31.863+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:32.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:31.864+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:32.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:31.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:31.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:31.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:32.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:31.859+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:31.910+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:31.910+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:31.910+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:32.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:31.911+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:33.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:32.816+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:33.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:32.816+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:33.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:32.816+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:33.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:32.817+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:32.852+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:32.852+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:32.852+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:32.853+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:33.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:32.959+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:33.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:32.959+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:33.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:32.959+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:33.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:32.960+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:34.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:33.831+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:34.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:33.831+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:34.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:33.831+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:34.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:33.832+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:33.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:34.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:33.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:34.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:33.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:34.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:33.958+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:34.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:33.863+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:34.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:33.863+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:34.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:33.863+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:34.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:33.863+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:35.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:34.905+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:35.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:34.905+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:35.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:34.905+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:35.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:34.906+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:35.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:34.813+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:35.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:34.813+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:35.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:34.813+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:35.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:34.814+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:35.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:34.965+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:35.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:34.965+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:35.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:34.965+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:35.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:34.966+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:36.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:35.908+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:36.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:35.908+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:36.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:35.908+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:36.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:35.908+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:36.011+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:36.011+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:36.011+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:36.012+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:36.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:35.819+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:36.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:35.819+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:36.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:35.819+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:36.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:35.820+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:36.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:51:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:51:36.246+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 35134 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:51:37.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:36.861+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:37.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:36.861+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:37.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:36.861+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:37.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:36.862+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:37.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:36.920+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:37.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:36.920+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:37.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:36.920+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:37.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:36.920+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:37.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:37.026+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:37.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:37.026+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:37.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:37.026+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:37.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:37.026+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:38.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:37.902+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:38.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:37.902+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:38.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:37.902+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:38.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:37.902+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:38.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:37.900+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:38.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:37.900+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:38.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:37.900+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:38.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:37.901+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:38.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:38.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:38.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:38.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:38.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:38.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:38.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:38.001+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:39.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:38.889+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:39.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:38.889+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:39.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:38.889+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:39.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:38.889+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:39.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:38.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:39.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:38.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:39.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:38.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:39.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:38.867+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:39.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:39.039+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:39.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:39.039+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:39.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:39.039+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:39.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:39.040+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:40.226 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:39.932+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:40.227 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:39.932+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:40.227 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:39.932+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:40.227 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:39.933+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:40.227 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:39.855+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:40.228 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:39.855+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:40.228 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:39.855+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:40.228 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:39.856+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:40.228 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:40.080+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:40.228 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:40.080+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:40.228 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:40.080+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:40.228 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:40.081+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:41.250 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:40.872+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:41.251 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:40.872+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:41.251 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:40.872+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:41.251 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:40.873+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:41.251 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:40.918+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:41.251 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:40.918+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:41.252 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:40.918+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:41.252 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:40.919+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:41.252 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:41.061+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:41.252 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:41.061+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:41.252 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:41.061+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:41.252 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:41.062+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:41.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:51:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:51:41.250+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 35224 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:51:42.035 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:41.881+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:42.036 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:41.881+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:42.036 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:41.881+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:42.036 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:41.882+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:42.036 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:41.865+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:42.037 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:41.865+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:42.037 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:41.865+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:42.037 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:41.866+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:42.037 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:42.035+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:42.037 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:42.035+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:42.037 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:42.035+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:42.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:42.036+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:43.069+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:43.069+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:43.069+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:43.070+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:42.877+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:42.877+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:42.877+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:43.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:42.878+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:43.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:42.893+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:43.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:42.893+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:43.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:42.893+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:43.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:42.894+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:44.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:43.903+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:44.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:43.903+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:44.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:43.903+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:44.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:43.904+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:44.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:43.853+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:44.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:43.853+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:44.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:43.853+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:44.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:43.854+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:44.043+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:44.043+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:44.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:44.043+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:44.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:44.044+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:45.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:44.933+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:45.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:44.933+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:45.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:44.933+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:45.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:44.933+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:45.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:44.872+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:45.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:44.872+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:45.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:44.872+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:45.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:44.873+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:45.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:45.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:45.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:45.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:45.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:45.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:45.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:45.000+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:46.255 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:45.951+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:46.255 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:45.951+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:46.255 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:45.951+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:46.255 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:45.952+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:46.256 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:46.008+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:46.256 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:46.008+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:46.256 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:46.008+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:46.256 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:46.009+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:46.256 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:45.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:46.257 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:45.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:46.257 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:45.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:46.257 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:45.868+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:46.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:51:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:51:46.254+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 35316 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:51:47.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:46.996+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:47.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:46.996+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:47.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:46.996+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:47.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:46.996+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:47.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:47.053+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:47.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:47.053+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:47.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:47.053+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:47.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:47.053+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:47.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:46.915+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:47.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:46.915+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:47.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:46.915+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:47.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:46.916+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:48.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:48.044+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:48.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:48.044+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:48.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:48.044+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:48.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:48.044+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:48.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:48.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:48.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:48.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:48.033+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:48.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:47.944+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:48.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:47.944+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:48.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:47.944+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:48.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:47.945+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:49.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:49.087+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:49.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:49.087+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:49.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:49.087+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:49.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:49.088+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:49.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:48.921+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:49.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:48.921+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:49.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:48.921+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:49.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:48.921+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:49.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:49.027+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:49.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:49.027+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:49.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:49.027+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:49.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:49.028+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:50.228 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:50.046+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:50.228 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:50.046+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:50.228 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:50.046+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:50.228 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:50.047+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:50.229 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:50.076+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:50.229 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:50.076+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:50.229 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:50.076+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:50.229 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:50.076+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:50.229 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:49.913+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:50.229 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:49.913+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:50.230 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:49.913+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:50.230 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:49.914+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:51.260 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:51.065+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:51.260 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:51.065+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:51.260 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:51.065+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:51.260 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:51.066+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:51.261 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:50.956+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:51.261 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:50.956+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:51.261 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:50.956+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:51.261 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:50.957+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:51.261 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:51.040+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:51.261 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:51.040+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:51.261 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:51.040+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:51.262 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:51.040+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:51.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:51:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:51:51.260+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 35407 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:51:52.057 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:51.911+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:52.058 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:51.911+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:52.058 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:51.911+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:52.058 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:51.912+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:52.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:52.058+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:52.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:52.058+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:52.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:52.058+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:52.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:52.059+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:52.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:52.073+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:52.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:52.073+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:52.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:52.073+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:52.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:52.073+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:53.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:53.067+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:53.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:53.067+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:53.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:53.067+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:53.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:53.068+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:53.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:53.055+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:53.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:53.055+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:53.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:53.055+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:53.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:53.056+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:53.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:52.933+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:53.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:52.933+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:53.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:52.933+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:53.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:52.934+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:54.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:54.081+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:54.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:54.081+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:54.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:54.081+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:54.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:54.082+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:54.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:53.901+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:54.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:53.901+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:54.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:53.901+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:54.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:53.902+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:54.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:54.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:54.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:54.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:54.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:54.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:54.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:54.106+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:55.130 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:54.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:55.130 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:54.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:55.130 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:54.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:55.130 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:54.866+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:55.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:55.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:55.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:55.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:55.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:55.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:55.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:55.137+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:55.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:55.130+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:55.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:55.130+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:55.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:55.130+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:55.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:55.132+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:56.265 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:56.086+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:56.265 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:56.086+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:56.266 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:56.086+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:56.266 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:56.087+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:56.266 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:55.905+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:56.266 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:55.905+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:56.266 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:55.905+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:56.267 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:55.906+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:56.267 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:56.101+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:56.267 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:56.101+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:56.267 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:56.101+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:56.267 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:56.101+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:56.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:51:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:51:56.264+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 35499 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:51:57.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:56.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:57.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:56.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:57.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:56.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:57.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:56.867+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:57.101+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:57.101+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:57.101+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:57.101+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:57.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:57.065+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:57.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:57.065+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:57.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:57.065+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:57.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:57.066+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:58.092+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:58.092+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:58.092+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:58.092+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:58.066+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:58.066+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:58.066+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:58.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:58.067+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:57.869+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:58.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:57.869+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:58.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:57.869+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:58.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:57.870+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:59.113 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:58.842+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:59.113 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:58.842+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:59.113 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:58.842+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:51:59.113 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:58.843+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:51:59.114 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:59.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:59.114 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:59.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:59.114 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:59.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:51:59.114 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:51:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:51:59.052+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:51:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:59.112+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:59.112+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:59.112+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:51:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:51:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:51:59.113+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:00.161 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:59.880+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:00.161 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:59.880+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:00.161 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:59.880+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:00.161 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:51:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:51:59.881+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:00.162 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:00.016+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:00.162 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:00.016+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:00.162 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:00.016+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:00.162 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:00.017+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:00.160+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:00.160+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:00.160+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:00.161+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:01.186 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:01.019+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:01.186 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:01.019+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:01.186 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:01.019+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:01.186 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:01.019+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:01.187 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:00.885+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:01.187 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:00.885+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:01.187 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:00.885+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:01.187 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:00.886+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:01.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:52:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:52:01.267+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 35589 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:52:01.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:01.185+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:01.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:01.185+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:01.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:01.185+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:01.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:01.186+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:02.025 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:01.928+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:02.026 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:01.928+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:02.026 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:01.928+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:02.026 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:01.928+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:02.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:02.025+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:02.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:02.025+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:02.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:02.025+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:02.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:02.026+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:02.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:02.201+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:02.201+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:02.201+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:02.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:02.202+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:03.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:03.209+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:03.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:03.209+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:03.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:03.209+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:03.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:03.210+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:02.977+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:02.977+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:02.977+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:03.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:02.977+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:03.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:03.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:03.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:03.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:03.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:03.051+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:04.254 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:03.958+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:04.255 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:03.958+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:04.255 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:03.958+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:04.255 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:03.958+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:04.255 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:04.059+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:04.255 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:04.059+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:04.255 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:04.059+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:04.256 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:04.060+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:04.254+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:04.254+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:04.254+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:04.255+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:05.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:04.987+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:05.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:04.987+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:05.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:04.987+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:05.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:04.987+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:05.272 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:05.087+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:05.272 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:05.087+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:05.272 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:05.087+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:05.273 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:05.088+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:05.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:05.271+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:05.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:05.271+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:05.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:05.271+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:05.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:05.272+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:06.295 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:52:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:52:06.272+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 35681 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:52:06.295 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:06.034+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:06.295 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:06.034+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:06.295 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:06.034+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:06.295 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:06.035+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:06.296 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:06.129+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:06.296 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:06.129+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:06.296 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:06.129+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:06.296 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:06.130+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:06.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:06.319+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:06.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:06.319+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:06.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:06.319+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:06.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:06.321+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:07.297 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:07.029+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:07.298 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:07.029+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:07.298 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:07.029+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:07.298 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:07.030+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:07.298 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:07.118+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:07.298 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:07.118+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:07.298 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:07.118+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:07.299 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:07.119+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:07.297+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:07.297+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:07.297+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:07.298+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:08.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:08.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:08.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:08.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:08.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:08.291+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:08.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:08.291+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:08.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:08.050+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:08.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:08.050+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:08.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:08.050+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:08.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:08.050+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:08.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:08.124+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:08.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:08.124+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:08.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:08.124+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:08.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:08.125+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:09.283 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:09.003+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:09.283 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:09.003+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:09.284 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:09.003+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:09.284 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:09.004+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:09.284 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:09.174+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:09.284 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:09.174+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:09.284 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:09.174+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:09.284 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:09.175+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:09.283+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:09.283+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:09.283+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:09.284+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:10.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:09.965+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:10.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:09.965+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:10.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:09.965+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:10.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:09.966+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:10.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:10.223+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:10.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:10.223+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:10.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:10.223+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:10.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:10.224+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:10.292+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:10.292+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:10.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:10.292+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:10.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:10.292+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:11.242 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:10.962+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:11.242 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:10.962+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:11.242 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:10.962+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:11.243 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:10.963+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:11.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:52:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:52:11.276+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 35772 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:52:11.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:11.265+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:11.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:11.265+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:11.265+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:11.265+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:11.241+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:11.241+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:11.242+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:11.242+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:12.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:11.933+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:12.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:11.933+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:12.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:11.933+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:12.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:11.933+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:12.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:12.249+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:12.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:12.249+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:12.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:12.249+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:12.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:12.249+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:12.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:12.286+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:12.286+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:12.286+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:12.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:12.286+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:13.211 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:12.944+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:13.211 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:12.944+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:13.211 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:12.944+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:13.212 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:12.944+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:13.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:13.312+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:13.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:13.312+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:13.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:13.312+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:13.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:13.313+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:13.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:13.211+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:13.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:13.211+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:13.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:13.211+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:13.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:13.211+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:14.214 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:13.921+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:14.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:13.921+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:14.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:13.921+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:14.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:13.922+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:14.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:14.313+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:14.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:14.313+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:14.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:14.313+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:14.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:14.314+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:14.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:14.214+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:14.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:14.214+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:14.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:14.214+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:14.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:14.215+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:15.254 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:14.909+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:15.254 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:14.909+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:15.255 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:14.909+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:15.255 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:14.910+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:15.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:15.331+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:15.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:15.331+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:15.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:15.331+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:15.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:15.332+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:15.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:15.254+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:15.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:15.254+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:15.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:15.254+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:15.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:15.255+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:16.248 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:15.954+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:16.248 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:15.954+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:16.249 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:15.954+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:16.249 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:15.955+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:16.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:52:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:52:16.281+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 35864 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:52:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:16.343+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:16.343+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:16.343+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:16.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:16.343+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:16.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:16.248+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:16.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:16.248+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:16.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:16.248+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:16.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:16.249+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:17.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:16.918+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:17.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:16.918+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:17.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:16.918+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:17.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:16.919+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:17.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:17.299+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:17.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:17.299+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:17.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:17.299+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:17.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:17.301+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:17.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:17.219+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:17.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:17.219+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:17.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:17.219+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:17.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:17.220+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:18.238 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:17.938+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:18.238 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:17.938+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:18.238 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:17.938+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:18.238 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:17.939+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:18.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:18.331+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:18.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:18.331+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:18.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:18.331+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:18.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:18.331+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:18.237+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:18.237+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:18.237+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:18.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:18.238+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:19.244 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:18.916+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:19.244 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:18.916+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:19.244 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:18.916+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:19.244 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:18.917+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:19.376+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:19.376+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:19.376+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:19.377+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:19.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:19.243+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:19.243+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:19.243+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:19.244+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:20.229 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:19.934+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:20.229 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:19.934+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:20.229 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:19.934+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:20.230 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:19.935+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:20.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:20.256+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:20.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:20.256+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:20.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:20.256+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:20.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:20.257+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:20.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:20.379+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:20.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:20.379+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:20.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:20.379+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:20.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:20.380+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:21.266 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:20.904+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:21.266 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:20.904+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:21.266 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:20.904+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:21.266 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:20.905+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:21.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:52:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:52:21.285+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 35954 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:52:21.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:21.423+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:21.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:21.423+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:21.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:21.423+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:21.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:21.424+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:21.266+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:21.266+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:21.266+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:21.266+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:22.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:21.869+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:21.869+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:21.869+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:21.870+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:22.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:22.312+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:22.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:22.312+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:22.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:22.312+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:22.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:22.313+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:22.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:22.473+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:22.473+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:22.473+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:22.474+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:23.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:22.820+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:22.820+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:22.820+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:22.821+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:23.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:23.447+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:23.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:23.447+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:23.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:23.447+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:23.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:23.448+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:23.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:23.358+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:23.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:23.358+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:23.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:23.358+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:23.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:23.358+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:24.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:23.867+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:24.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:23.867+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:24.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:23.867+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:24.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:23.868+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:24.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:24.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:24.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:24.400+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:24.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:24.380+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:24.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:24.380+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:24.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:24.380+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:24.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:24.381+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:25.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:24.881+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:25.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:24.881+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:25.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:24.881+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:25.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:24.882+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:25.430+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:25.430+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:25.430+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:25.431+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:25.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:25.360+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:25.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:25.360+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:25.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:25.360+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:25.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:25.361+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:26.290 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:25.888+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:26.291 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:25.888+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:26.291 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:25.888+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:26.291 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:25.889+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:26.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:52:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:52:26.290+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 36046 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:52:26.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:26.409+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:26.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:26.409+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:26.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:26.409+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:26.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:26.409+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:26.450+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:26.450+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:26.450+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:26.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:26.451+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:27.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:26.910+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:27.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:26.910+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:27.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:26.910+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:27.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:26.911+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:27.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:27.474+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:27.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:27.474+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:27.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:27.474+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:27.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:27.475+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:27.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:27.366+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:27.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:27.366+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:27.366+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:27.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:27.366+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:28.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:27.892+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:28.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:27.892+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:28.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:27.892+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:28.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:27.892+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:28.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:28.401+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:28.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:28.401+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:28.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:28.401+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:28.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:28.402+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:28.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:28.505+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:28.505+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:28.505+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:28.506+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:29.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:28.854+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:29.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:28.854+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:29.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:28.854+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:29.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:28.855+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:29.465+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:29.465+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:29.465+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:29.466+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:29.354+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:29.354+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:29.354+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:29.355+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:30.230 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:29.848+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:30.230 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:29.848+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:30.230 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:29.848+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:30.230 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:29.848+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:30.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:30.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:30.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:30.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:30.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:30.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:30.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:30.479+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:30.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:30.345+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:30.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:30.345+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:30.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:30.345+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:30.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:30.346+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:31.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:30.802+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:31.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:30.802+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:31.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:30.802+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:31.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:30.803+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:31.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:52:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:52:31.294+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 36137 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:52:31.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:31.363+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:31.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:31.363+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:31.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:31.363+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:31.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:31.364+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:31.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:31.484+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:31.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:31.484+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:31.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:31.484+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:31.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:31.485+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:32.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:31.756+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:32.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:31.756+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:32.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:31.756+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:32.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:31.757+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:32.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:32.504+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:32.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:32.504+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:32.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:32.504+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:32.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:32.504+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:32.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:32.412+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:32.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:32.412+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:32.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:32.412+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:32.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:32.413+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:33.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:32.724+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:33.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:32.724+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:33.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:32.724+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:33.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:32.725+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:33.775 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:33.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:33.775 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:33.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:33.775 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:33.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:33.775 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:33.479+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:33.776 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:33.384+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:33.776 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:33.384+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:33.776 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:33.384+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:33.776 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:33.385+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:34.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:33.774+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:34.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:33.774+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:34.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:33.774+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:34.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:33.775+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:34.755 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:34.464+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:34.755 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:34.464+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:34.755 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:34.464+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:34.756 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:34.465+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:34.756 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:34.384+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:34.756 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:34.384+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:34.756 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:34.384+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:34.756 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:34.385+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:35.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:34.755+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:35.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:34.755+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:35.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:34.755+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:35.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:34.755+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:35.706 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:35.475+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:35.707 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:35.475+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:35.707 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:35.475+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:35.707 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:35.476+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:35.707 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:35.383+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:35.707 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:35.383+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:35.707 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:35.383+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:35.707 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:35.383+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:36.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:35.706+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:36.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:35.706+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:36.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:35.706+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:36.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:35.707+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:36.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:52:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:52:36.299+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 36229 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:52:36.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:36.424+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:36.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:36.424+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:36.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:36.424+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:36.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:36.424+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:36.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:36.497+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:36.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:36.497+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:36.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:36.497+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:36.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:36.498+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:37.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:36.721+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:37.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:36.721+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:37.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:36.721+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:37.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:36.721+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:37.731 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:37.515+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:37.731 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:37.516+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:37.732 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:37.516+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:37.732 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:37.517+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:37.732 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:37.378+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:37.732 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:37.378+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:37.732 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:37.378+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:37.733 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:37.378+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:38.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:37.731+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:38.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:37.731+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:38.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:37.731+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:38.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:37.731+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:38.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:38.489+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:38.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:38.489+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:38.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:38.489+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:38.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:38.490+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:38.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:38.330+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:38.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:38.330+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:38.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:38.330+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:38.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:38.331+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:39.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:38.747+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:39.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:38.747+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:39.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:38.747+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:39.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:38.749+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:39.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:39.291+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:39.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:39.291+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:39.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:39.291+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:39.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:39.291+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:39.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:39.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:39.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:39.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:39.479+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:40.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:39.699+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:40.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:39.699+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:40.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:39.700+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:40.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:39.701+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:40.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:40.255+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:40.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:40.255+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:40.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:40.255+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:40.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:40.256+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:40.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:40.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:40.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:40.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:40.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:40.433+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:40.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:40.434+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:41.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:40.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:41.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:40.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:41.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:40.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:41.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:40.745+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:41.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:52:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:52:41.303+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 36319 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:52:41.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:41.427+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:41.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:41.427+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:41.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:41.427+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:41.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:41.428+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:41.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:41.238+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:41.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:41.238+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:41.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:41.238+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:41.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:41.239+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:42.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:41.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:42.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:41.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:42.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:41.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:42.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:41.770+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:42.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:42.411+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:42.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:42.411+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:42.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:42.411+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:42.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:42.412+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:42.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:42.252+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:42.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:42.252+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:42.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:42.252+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:42.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:42.252+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:43.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:42.745+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:43.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:42.745+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:43.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:42.745+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:43.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:42.746+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:43.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:43.291+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:43.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:43.291+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:43.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:43.291+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:43.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:43.292+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:43.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:43.395+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:43.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:43.395+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:43.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:43.395+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:43.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:43.396+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:44.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:43.733+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:44.149 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:43.733+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:44.149 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:43.733+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:44.149 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:43.734+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:44.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:44.403+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:44.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:44.403+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:44.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:44.403+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:44.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:44.404+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:44.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:44.324+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:44.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:44.324+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:44.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:44.324+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:44.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:44.324+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:45.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:44.740+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:45.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:44.740+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:45.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:44.740+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:45.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:44.741+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:45.770 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:45.413+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:45.770 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:45.413+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:45.770 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:45.413+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:45.770 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:45.414+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:45.771 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:45.370+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:45.771 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:45.371+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:45.771 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:45.371+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:45.771 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:45.371+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:46.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:45.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:46.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:45.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:46.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:45.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:46.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:45.770+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:46.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:52:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:52:46.308+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 36411 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:52:46.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:46.368+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:46.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:46.368+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:46.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:46.368+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:46.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:46.368+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:46.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:46.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:46.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:46.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:46.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:46.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:46.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:46.450+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:47.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:46.806+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:47.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:46.806+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:47.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:46.806+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:47.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:46.807+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:47.768 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:47.441+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:47.769 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:47.441+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:47.769 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:47.441+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:47.769 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:47.442+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:47.769 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:47.392+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:47.769 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:47.393+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:47.769 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:47.393+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:47.770 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:47.393+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:48.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:47.768+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:48.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:47.768+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:48.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:47.768+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:48.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:47.769+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:48.763 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:48.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:48.764 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:48.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:48.764 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:48.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:48.764 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:48.406+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:48.764 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:48.350+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:48.764 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:48.350+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:48.765 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:48.350+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:48.765 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:48.350+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:49.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:48.763+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:48.763+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:48.763+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:48.764+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:49.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:49.419+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:49.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:49.419+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:49.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:49.419+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:49.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:49.420+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:49.399+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:49.399+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:49.399+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:49.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:49.400+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:50.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:49.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:49.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:49.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:49.795+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:50.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:50.466+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:50.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:50.466+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:50.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:50.466+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:50.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:50.467+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:50.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:50.381+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:50.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:50.381+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:50.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:50.381+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:50.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:50.382+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:51.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:50.765+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:51.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:50.765+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:51.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:50.765+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:51.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:50.765+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:51.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:52:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:52:51.312+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 36502 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:52:51.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:51.400+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:51.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:51.400+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:51.400+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:51.401+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:51.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:51.460+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:51.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:51.460+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:51.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:51.460+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:51.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:51.460+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:52.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:51.756+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:52.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:51.756+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:52.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:51.756+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:52.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:51.757+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:52.713 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:52.381+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:52.713 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:52.381+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:52.713 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:52.381+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:52.713 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:52.382+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:52.714 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:52.416+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:52.714 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:52.416+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:52.714 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:52.416+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:52.714 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:52.417+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:53.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:52.713+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:53.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:52.713+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:53.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:52.713+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:53.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:52.714+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:53.752 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:53.370+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:53.752 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:53.370+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:53.752 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:53.370+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:53.753 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:53.371+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:53.753 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:53.350+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:53.753 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:53.350+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:53.753 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:53.350+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:53.753 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:53.351+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:54.051 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:53.752+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:54.051 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:53.752+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:54.052 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:53.752+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:54.052 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:53.752+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:54.791 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:54.369+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:54.792 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:54.369+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:54.792 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:54.369+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:54.792 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:54.370+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:54.792 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:54.378+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:54.792 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:54.378+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:54.792 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:54.378+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:54.792 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:54.378+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:55.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:54.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:55.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:54.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:55.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:54.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:55.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:54.792+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:55.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:55.335+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:55.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:55.335+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:55.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:55.335+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:55.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:55.336+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:55.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:55.346+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:55.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:55.346+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:55.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:55.346+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:55.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:55.347+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:56.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:55.770+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:56.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:55.770+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:56.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:55.770+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:56.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:55.770+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:56.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:52:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:52:56.317+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 36595 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:52:56.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:56.355+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:56.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:56.356+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:56.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:56.356+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:56.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:56.356+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:56.302+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:56.302+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:56.302+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:56.302+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:57.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:56.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:57.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:56.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:57.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:56.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:57.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:56.745+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:57.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:57.406+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:57.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:57.406+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:57.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:57.406+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:57.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:57.407+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:57.324+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:57.324+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:57.324+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:57.325+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:58.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:57.752+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:58.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:57.752+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:58.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:57.752+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:58.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:57.753+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:58.791 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:58.392+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:58.791 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:58.392+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:58.791 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:58.392+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:58.792 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:58.393+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:58.792 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:58.342+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:58.792 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:58.342+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:58.792 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:58.342+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:58.792 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:58.343+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:59.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:58.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:59.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:58.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:59.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:58.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:52:59.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:58.792+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:52:59.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:59.346+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:59.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:59.346+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:59.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:59.346+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:52:59.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:52:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:52:59.347+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:52:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:59.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:59.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:59.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:52:59.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:52:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:52:59.401+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:00.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:59.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:00.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:59.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:00.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:59.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:00.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:52:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:52:59.795+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:00.389+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:00.389+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:00.389+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:00.390+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:00.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:00.312+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:00.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:00.312+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:00.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:00.312+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:00.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:00.313+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:01.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:00.800+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:01.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:00.800+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:01.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:00.800+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:01.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:00.801+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:01.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:53:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:53:01.321+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 36685 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:53:01.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:01.364+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:01.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:01.364+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:01.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:01.364+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:01.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:01.365+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:01.266+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:01.266+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:01.266+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:01.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:01.267+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:02.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:01.770+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:02.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:01.770+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:02.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:01.770+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:02.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:01.771+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:02.358 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:02.308+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:02.358 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:02.308+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:02.358 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:02.308+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:02.358 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:02.308+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:02.736 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:02.358+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:02.736 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:02.358+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:02.736 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:02.358+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:02.737 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:02.358+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:03.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:02.736+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:03.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:02.736+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:03.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:02.736+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:03.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:02.737+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:03.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:03.322+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:03.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:03.322+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:03.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:03.322+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:03.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:03.323+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:03.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:03.287+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:03.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:03.287+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:03.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:03.287+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:03.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:03.288+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:04.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:03.726+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:04.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:03.726+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:04.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:03.726+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:04.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:03.727+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:04.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:04.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:04.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:04.341+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:04.292+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:04.292+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:04.292+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:04.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:04.292+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:04.700+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:04.700+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:04.700+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:04.701+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:05.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:05.299+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:05.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:05.299+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:05.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:05.299+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:05.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:05.300+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:05.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:05.321+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:05.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:05.321+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:05.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:05.321+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:05.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:05.321+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:06.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:05.676+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:06.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:05.676+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:06.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:05.676+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:06.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:05.677+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:06.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:53:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:53:06.326+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 36777 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:53:06.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:06.309+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:06.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:06.309+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:06.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:06.309+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:06.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:06.310+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:06.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:06.295+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:06.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:06.295+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:06.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:06.295+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:06.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:06.295+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:07.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:06.663+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:07.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:06.663+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:07.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:06.663+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:07.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:06.664+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:07.342+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:07.342+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:07.342+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:07.342+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:07.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:07.322+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:07.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:07.322+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:07.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:07.322+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:07.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:07.323+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:08.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:07.662+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:08.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:07.662+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:08.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:07.662+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:08.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:07.663+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:08.652 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:08.366+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:08.652 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:08.366+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:08.652 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:08.366+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:08.652 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:08.367+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:08.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:08.351+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:08.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:08.351+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:08.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:08.351+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:08.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:08.351+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:09.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:08.653+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:09.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:08.653+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:09.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:08.653+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:09.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:08.653+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:09.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:09.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:09.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:09.362+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:09.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:09.344+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:09.344+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:09.344+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:09.344+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:10.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:09.678+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:10.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:09.678+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:10.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:09.678+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:10.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:09.678+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:10.345+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:10.345+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:10.345+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:10.346+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:10.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:10.351+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:10.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:10.351+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:10.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:10.351+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:10.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:10.351+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:10.670+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:10.670+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:10.670+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:11.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:10.671+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:11.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:53:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:53:11.332+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 36868 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:53:11.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:11.301+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:11.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:11.301+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:11.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:11.301+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:11.303+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:11.368+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:11.368+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:11.368+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:11.368+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:12.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:11.721+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:12.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:11.721+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:12.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:11.721+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:12.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:11.721+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:12.734 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:12.407+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:12.734 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:12.407+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:12.734 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:12.407+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:12.734 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:12.408+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:12.734 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:12.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:12.735 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:12.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:12.735 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:12.340+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:12.735 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:12.341+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:13.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:12.734+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:13.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:12.734+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:13.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:12.734+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:13.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:12.735+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:13.769 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:13.390+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:13.769 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:13.390+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:13.769 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:13.390+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:13.769 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:13.390+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:13.769 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:13.363+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:13.769 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:13.363+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:13.769 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:13.363+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:13.770 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:13.363+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:13.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:13.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:13.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:13.770+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:14.738 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:14.366+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:14.739 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:14.366+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:14.739 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:14.366+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:14.739 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:14.367+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:14.739 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:14.356+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:14.739 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:14.356+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:14.739 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:14.356+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:14.740 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:14.357+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:15.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:14.739+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:15.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:14.739+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:15.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:14.739+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:15.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:14.740+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:15.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:15.410+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:15.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:15.410+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:15.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:15.410+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:15.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:15.411+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:15.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:15.337+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:15.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:15.337+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:15.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:15.337+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:15.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:15.337+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:16.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:15.777+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:16.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:15.777+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:16.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:15.777+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:16.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:15.777+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:16.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:53:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:53:16.336+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 36960 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:53:16.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:16.378+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:16.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:16.378+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:16.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:16.378+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:16.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:16.380+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:16.309+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:16.309+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:16.309+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:16.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:16.310+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:17.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:16.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:17.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:16.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:17.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:16.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:17.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:16.770+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:17.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:17.261+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:17.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:17.261+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:17.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:17.261+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:17.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:17.261+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:17.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:17.387+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:17.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:17.387+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:17.387+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:17.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:17.388+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:18.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:17.770+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:18.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:17.770+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:18.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:17.770+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:18.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:17.771+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:18.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:18.237+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:18.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:18.237+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:18.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:18.237+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:18.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:18.238+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:18.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:18.384+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:18.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:18.384+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:18.384+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:18.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:18.385+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:19.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:18.726+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:19.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:18.726+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:19.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:18.726+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:19.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:18.727+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:19.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:19.393+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:19.393+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:19.393+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:19.394+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:19.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:19.232+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:19.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:19.232+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:19.232+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:19.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:19.233+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:20.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:19.695+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:20.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:19.695+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:20.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:19.695+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:20.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:19.695+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:20.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:20.278+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:20.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:20.278+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:20.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:20.278+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:20.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:20.278+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:20.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:20.442+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:20.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:20.442+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:20.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:20.442+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:20.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:20.443+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:21.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:20.745+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:21.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:20.745+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:21.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:20.745+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:21.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:20.746+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:21.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:53:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:53:21.340+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 37050 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:53:21.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:21.302+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:21.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:21.302+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:21.302+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:21.303+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:21.459+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:21.459+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:21.459+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:21.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:21.460+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:21.765+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:21.765+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:21.765+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:21.766+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:22.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:22.308+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:22.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:22.308+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:22.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:22.308+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:22.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:22.308+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:22.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:22.443+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:22.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:22.443+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:22.443+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:22.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:22.444+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:23.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:22.768+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:22.768+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:22.768+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:22.769+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:23.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:23.309+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:23.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:23.309+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:23.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:23.309+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:23.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:23.310+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:23.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:23.452+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:23.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:23.452+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:23.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:23.452+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:23.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:23.453+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:24.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:23.756+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:24.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:23.756+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:24.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:23.756+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:24.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:23.756+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:24.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:24.298+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:24.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:24.298+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:24.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:24.298+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:24.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:24.299+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:24.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:24.457+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:24.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:24.457+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:24.457+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:24.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:24.458+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:25.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:24.734+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:25.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:24.734+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:25.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:24.734+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:25.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:24.734+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:25.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:25.412+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:25.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:25.412+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:25.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:25.412+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:25.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:25.413+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:25.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:25.327+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:25.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:25.327+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:25.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:25.327+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:25.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:25.328+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:26.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:25.759+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:26.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:25.759+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:26.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:25.759+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:26.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:25.760+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:26.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:53:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:53:26.345+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 37142 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:53:26.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:26.446+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:26.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:26.446+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:26.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:26.446+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:26.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:26.447+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:26.322+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:26.322+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:26.322+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:26.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:26.322+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:27.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:26.781+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:27.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:26.781+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:27.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:26.781+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:27.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:26.782+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:27.784 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:27.494+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:27.784 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:27.494+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:27.784 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:27.494+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:27.784 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:27.495+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:27.785 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:27.350+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:27.785 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:27.350+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:27.785 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:27.350+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:27.785 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:27.351+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:28.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:27.784+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:27.784+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:27.784+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:27.784+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:28.738 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:28.346+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:28.738 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:28.346+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:28.738 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:28.346+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:28.738 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:28.347+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:28.739 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:28.545+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:28.739 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:28.545+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:28.739 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:28.545+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:28.739 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:28.545+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:29.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:28.737+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:29.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:28.737+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:29.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:28.737+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:29.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:28.738+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:29.510 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:29.352+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:29.510 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:29.352+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:29.511 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:29.352+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:29.511 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:29.353+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:29.762 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:29.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:29.762 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:29.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:29.762 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:29.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:29.762 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:29.511+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:30.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:29.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:30.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:29.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:30.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:29.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:30.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:29.762+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:30.518 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:30.395+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:30.518 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:30.396+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:30.518 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:30.396+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:30.519 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:30.396+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:30.805 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:30.518+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:30.805 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:30.518+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:30.805 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:30.518+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:30.806 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:30.519+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:31.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:30.805+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:31.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:30.805+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:31.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:30.805+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:31.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:30.806+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:31.774 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:53:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:53:31.349+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 37233 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:53:31.775 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:31.361+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:31.775 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:31.361+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:31.775 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:31.361+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:31.775 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:31.362+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:31.775 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:31.558+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:31.776 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:31.558+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:31.776 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:31.558+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:31.776 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:31.559+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:32.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:31.801+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:32.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:31.801+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:32.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:31.801+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:32.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:31.802+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:32.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:32.329+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:32.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:32.329+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:32.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:32.329+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:32.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:32.329+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:33.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:32.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:33.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:32.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:33.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:32.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:33.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:32.609+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:33.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:32.785+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:33.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:32.785+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:32.785+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:32.786+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:33.836 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:33.589+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:33.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:33.589+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:33.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:33.589+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:33.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:33.590+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:33.837 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:33.373+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:33.837 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:33.373+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:33.837 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:33.373+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:33.837 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:33.373+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:34.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:33.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:34.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:33.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:34.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:33.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:34.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:33.837+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:34.837 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:34.394+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:34.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:34.394+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:34.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:34.394+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:34.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:34.395+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:34.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:34.552+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:34.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:34.552+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:34.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:34.552+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:34.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:34.552+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:35.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:34.865+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:35.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:34.865+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:35.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:34.865+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:35.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:34.866+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:35.831 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:35.598+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:35.831 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:35.598+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:35.831 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:35.598+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:35.831 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:35.598+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:35.832 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:35.397+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:35.832 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:35.397+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:35.832 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:35.397+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:35.832 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:35.398+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:36.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:35.831+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:36.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:35.831+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:36.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:35.831+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:36.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:35.832+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:36.622 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:53:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:53:36.354+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 37325 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:53:36.622 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:36.434+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:36.623 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:36.434+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:36.623 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:36.434+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:36.623 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:36.435+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:37.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:36.803+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:37.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:36.803+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:37.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:36.803+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:37.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:36.804+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:37.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:36.622+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:37.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:36.622+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:37.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:36.622+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:37.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:36.623+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:37.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:37.477+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:37.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:37.477+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:37.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:37.477+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:37.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:37.478+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:37.606+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:37.606+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:37.606+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:37.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:37.606+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:38.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:37.839+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:38.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:37.839+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:38.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:37.839+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:38.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:37.840+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:38.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:38.576+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:38.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:38.576+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:38.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:38.576+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:38.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:38.577+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:38.822 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:38.467+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:38.822 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:38.467+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:38.822 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:38.467+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:38.822 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:38.468+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:39.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:38.821+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:39.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:38.821+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:39.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:38.821+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:39.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:38.822+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:39.577 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:39.465+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:39.577 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:39.465+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:39.578 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:39.465+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:39.578 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:39.465+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:39.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:39.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:39.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:39.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:39.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:39.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:39.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:39.791+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:39.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:39.577+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:39.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:39.577+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:39.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:39.577+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:39.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:39.578+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:40.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:40.440+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:40.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:40.440+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:40.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:40.440+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:40.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:40.441+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:41.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:40.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:41.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:40.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:41.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:40.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:41.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:40.763+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:41.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:40.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:41.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:40.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:41.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:40.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:41.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:40.596+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:41.619 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:53:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:53:41.358+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 37410 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:53:41.619 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:41.463+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:41.619 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:41.463+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:41.620 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:41.463+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:41.620 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:41.464+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:42.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:41.770+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:42.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:41.770+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:42.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:41.770+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:42.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:41.771+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:42.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:41.619+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:42.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:41.619+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:42.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:41.619+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:42.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:41.620+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:42.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:42.635+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:42.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:42.635+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:42.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:42.635+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:42.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:42.636+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:42.798 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:42.510+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:42.798 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:42.510+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:42.798 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:42.510+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:42.798 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:42.510+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:43.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:42.797+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:43.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:42.797+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:43.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:42.797+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:43.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:42.798+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:43.836 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:43.603+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:43.836 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:43.603+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:43.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:43.603+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:43.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:43.604+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:43.837 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:43.475+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:43.837 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:43.475+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:43.837 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:43.475+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:43.837 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:43.475+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:44.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:43.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:44.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:43.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:44.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:43.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:44.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:43.837+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:44.602+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:44.602+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:44.602+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:44.603+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:44.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:44.517+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:44.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:44.517+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:44.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:44.517+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:44.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:44.517+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:45.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:44.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:45.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:44.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:45.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:44.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:45.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:44.848+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:45.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:45.518+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:45.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:45.518+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:45.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:45.518+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:45.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:45.518+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:45.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:45.557+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:45.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:45.557+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:45.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:45.557+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:45.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:45.558+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:46.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:45.874+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:46.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:45.874+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:46.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:45.874+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:46.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:45.875+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:46.833 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:53:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:53:46.363+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 37500 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:53:46.833 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:46.603+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:46.833 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:46.603+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:46.834 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:46.603+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:46.834 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:46.604+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:46.834 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:46.484+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:46.834 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:46.484+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:46.834 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:46.484+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:46.834 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:46.485+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:47.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:46.833+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:47.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:46.833+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:47.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:46.833+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:47.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:46.834+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:47.837 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:47.526+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:47.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:47.526+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:47.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:47.526+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:47.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:47.527+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:47.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:47.613+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:47.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:47.613+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:47.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:47.613+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:47.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:47.613+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:48.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:47.855+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:48.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:47.855+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:48.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:47.855+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:48.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:47.855+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:48.830 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:48.525+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:48.831 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:48.525+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:48.831 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:48.525+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:48.831 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:48.526+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:48.831 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:48.657+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:48.831 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:48.657+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:48.832 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:48.657+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:48.832 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:48.658+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:49.091 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:48.830+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:49.091 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:48.830+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:49.091 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:48.830+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:49.091 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:48.831+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:49.819 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:49.620+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:49.820 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:49.620+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:49.820 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:49.620+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:49.820 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:49.620+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:49.820 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:49.551+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:49.820 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:49.551+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:49.820 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:49.551+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:49.821 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:49.552+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:50.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:49.819+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:49.819+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:49.819+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:49.820+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:50.771 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:50.610+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:50.772 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:50.610+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:50.772 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:50.610+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:50.772 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:50.611+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:50.772 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:50.517+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:50.772 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:50.517+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:50.772 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:50.517+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:50.772 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:50.517+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:51.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:50.771+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:51.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:50.771+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:51.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:50.771+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:51.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:50.772+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:51.660 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:53:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:53:51.368+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 37588 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:53:51.660 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:51.484+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:51.660 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:51.484+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:51.660 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:51.484+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:51.661 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:51.485+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:52.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:51.817+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:52.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:51.817+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:52.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:51.817+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:52.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:51.818+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:52.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:51.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:51.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:51.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:51.661+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:52.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:52.637+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:52.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:52.637+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:52.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:52.637+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:52.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:52.638+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:52.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:52.502+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:52.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:52.502+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:52.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:52.502+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:52.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:52.503+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:53.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:52.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:53.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:52.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:53.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:52.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:53.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:52.859+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:53.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:53.686+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:53.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:53.686+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:53.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:53.686+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:53.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:53.687+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:53.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:53.501+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:53.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:53.501+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:53.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:53.501+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:53.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:53.502+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:54.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:53.845+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:54.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:53.845+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:54.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:53.845+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:54.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:53.846+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:54.694+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:54.694+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:54.694+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:54.694+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:54.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:54.466+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:54.466+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:54.466+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:54.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:54.467+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:55.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:54.880+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:55.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:54.880+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:55.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:54.880+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:55.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:54.880+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:55.724+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:55.724+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:55.724+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:55.724+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:55.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:55.516+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:55.516+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:55.516+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:55.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:55.517+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:56.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:55.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:56.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:55.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:56.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:55.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:56.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:55.859+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:56.708 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:53:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:53:56.372+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 37679 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:53:56.708 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:56.527+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:56.708 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:56.527+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:56.708 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:56.527+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:56.709 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:56.527+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:57.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:56.708+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:57.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:56.708+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:57.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:56.708+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:57.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:56.708+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:57.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:56.903+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:57.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:56.903+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:57.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:56.903+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:57.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:56.904+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:57.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:57.670+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:57.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:57.670+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:57.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:57.670+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:57.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:57.671+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:57.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:57.484+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:57.484+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:57.484+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:57.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:57.484+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:58.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:57.948+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:58.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:57.948+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:58.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:57.948+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:58.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:57.949+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:58.644+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:58.644+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:58.644+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:58.645+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:58.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:58.520+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:58.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:58.520+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:58.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:58.520+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:58.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:58.521+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:59.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:58.923+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:59.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:58.923+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:59.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:58.923+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:53:59.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:58.924+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:53:59.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:59.644+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:59.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:59.644+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:59.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:59.644+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:53:59.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:53:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:53:59.646+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:53:59.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:59.537+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:59.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:59.537+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:59.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:59.537+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:53:59.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:53:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:53:59.538+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:00.236 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:59.913+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:00.237 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:59.913+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:00.237 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:59.913+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:00.237 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:53:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:53:59.914+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:00.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:00.540+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:00.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:00.541+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:00.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:00.541+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:00.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:00.541+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:00.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:00.671+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:00.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:00.671+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:00.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:00.671+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:00.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:00.671+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:01.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:00.962+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:01.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:00.962+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:01.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:00.962+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:01.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:00.963+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:01.635 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:54:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:54:01.377+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 37771 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:54:01.635 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:01.547+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:01.636 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:01.547+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:01.636 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:01.547+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:01.636 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:01.547+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:01.980 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:01.717+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:01.980 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:01.717+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:01.980 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:01.717+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:01.980 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:01.718+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:02.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:01.979+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:02.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:01.979+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:02.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:01.979+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:02.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:01.980+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:02.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:02.527+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:02.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:02.527+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:02.704 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:02.527+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:02.704 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:02.528+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:03.001 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:02.703+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:03.001 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:02.703+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:03.001 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:02.703+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:03.001 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:02.704+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:03.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:03.000+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:03.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:03.000+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:03.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:03.000+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:03.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:03.001+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:03.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:03.734+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:03.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:03.734+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:03.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:03.734+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:03.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:03.734+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:03.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:03.514+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:03.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:03.514+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:03.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:03.514+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:03.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:03.515+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:04.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:04.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:04.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:04.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:04.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:04.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:04.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:04.021+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:04.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:04.699+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:04.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:04.699+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:04.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:04.699+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:04.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:04.699+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:04.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:04.493+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:04.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:04.493+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:04.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:04.493+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:04.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:04.494+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:05.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:05.039+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:05.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:05.039+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:05.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:05.039+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:05.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:05.040+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:05.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:05.734+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:05.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:05.734+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:05.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:05.734+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:05.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:05.735+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:05.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:05.497+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:05.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:05.497+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:05.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:05.497+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:05.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:05.499+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:06.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:06.043+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:06.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:06.043+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:06.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:06.043+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:06.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:06.043+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:06.766 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:54:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:54:06.382+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 37861 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:54:06.767 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:06.543+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:06.767 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:06.543+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:06.767 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:06.543+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:06.767 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:06.544+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:07.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:06.766+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:07.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:06.766+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:07.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:06.766+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:07.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:06.767+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:07.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:07.003+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:07.003+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:07.003+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:07.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:07.004+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:07.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:07.792+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:07.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:07.792+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:07.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:07.792+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:07.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:07.793+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:07.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:07.580+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:07.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:07.580+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:07.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:07.581+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:07.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:07.581+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:08.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:08.037+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:08.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:08.037+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:08.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:08.037+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:08.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:08.038+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:09.019 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:08.828+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:09.019 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:08.828+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:09.019 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:08.828+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:09.020 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:08.829+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:09.020 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:08.602+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:09.020 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:08.602+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:09.020 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:08.602+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:09.020 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:08.603+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:09.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:09.019+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:09.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:09.019+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:09.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:09.019+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:09.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:09.020+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:09.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:09.836+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:09.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:09.836+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:09.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:09.836+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:09.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:09.837+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:09.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:09.629+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:09.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:09.629+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:09.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:09.629+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:09.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:09.630+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:10.237 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:09.974+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:10.238 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:09.974+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:10.238 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:09.974+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:10.238 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:09.975+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:10.987 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:10.812+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:10.988 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:10.812+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:10.988 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:10.812+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:10.988 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:10.813+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:10.988 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:10.659+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:10.989 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:10.659+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:10.989 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:10.659+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:10.989 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:10.660+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:11.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:10.987+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:11.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:10.987+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:11.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:10.987+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:11.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:10.988+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:11.689 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:54:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:54:11.387+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 37954 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:54:12.007 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:11.801+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:12.007 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:11.801+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:12.007 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:11.801+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:12.007 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:11.802+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:12.007 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:11.689+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:12.008 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:11.689+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:12.008 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:11.689+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:12.008 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:11.690+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:12.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:12.007+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:12.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:12.007+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:12.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:12.007+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:12.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:12.008+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:13.033 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:12.832+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:13.034 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:12.832+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:13.034 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:12.832+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:13.034 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:12.833+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:13.034 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:12.680+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:13.034 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:12.680+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:13.034 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:12.680+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:13.035 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:12.681+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:13.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:13.033+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:13.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:13.033+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:13.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:13.033+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:13.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:13.034+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:14.064 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:13.822+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:14.064 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:13.822+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:14.064 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:13.822+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:14.065 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:13.823+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:14.065 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:13.701+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:14.065 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:13.701+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:14.065 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:13.701+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:14.065 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:13.701+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:14.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:14.064+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:14.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:14.064+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:14.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:14.064+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:14.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:14.065+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:15.029 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:14.691+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:15.030 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:14.691+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:15.030 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:14.691+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:15.030 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:14.692+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:15.030 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:14.804+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:15.030 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:14.804+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:15.031 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:14.804+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:15.031 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:14.805+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:15.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:15.029+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:15.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:15.029+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:15.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:15.029+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:15.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:15.030+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:16.036 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:15.798+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:16.036 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:15.798+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:16.036 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:15.798+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:16.036 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:15.798+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:16.037 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:15.692+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:16.037 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:15.692+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:16.037 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:15.692+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:16.037 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:15.693+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:16.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:16.035+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:16.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:16.035+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:16.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:16.035+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:16.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:16.036+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:16.697 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:54:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:54:16.392+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 38045 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:54:17.051 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:16.847+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:17.052 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:16.847+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:17.052 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:16.847+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:17.052 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:16.847+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:17.052 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:16.697+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:17.052 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:16.697+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:17.052 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:16.697+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:17.053 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:16.698+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:17.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:17.051+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:17.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:17.051+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:17.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:17.051+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:17.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:17.052+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:18.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:17.848+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:18.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:17.848+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:18.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:17.848+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:18.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:17.848+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:18.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:17.685+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:18.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:17.685+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:18.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:17.685+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:18.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:17.686+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:18.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:18.095+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:18.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:18.095+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:18.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:18.095+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:18.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:18.096+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:19.059 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:18.818+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:19.060 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:18.818+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:19.060 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:18.818+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:19.060 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:18.818+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:19.060 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:18.662+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:19.061 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:18.662+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:19.061 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:18.662+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:19.061 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:18.663+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:19.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:19.059+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:19.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:19.059+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:19.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:19.059+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:19.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:19.061+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:20.070 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:19.798+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:20.070 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:19.799+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:20.070 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:19.799+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:20.070 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:19.799+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:20.071 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:19.635+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:20.071 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:19.635+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:20.071 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:19.635+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:20.071 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:19.636+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:20.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:20.069+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:20.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:20.069+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:20.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:20.069+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:20.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:20.070+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:21.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:20.623+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:21.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:20.623+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:21.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:20.623+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:21.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:20.624+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:21.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:20.830+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:21.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:20.830+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:21.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:20.830+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:21.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:20.831+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:21.397 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:21.089+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:21.418 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:21.089+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:21.418 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:21.089+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:21.418 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:21.090+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:21.666 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:54:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:54:21.396+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 38137 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:54:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:21.665+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:21.665+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:21.665+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:21.666+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:22.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:21.858+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:22.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:21.858+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:22.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:21.858+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:22.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:21.859+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:22.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:22.139+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:22.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:22.139+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:22.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:22.139+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:22.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:22.140+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:23.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:22.813+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:23.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:22.813+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:23.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:22.813+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:23.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:22.813+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:22.674+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:23.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:22.674+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:23.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:22.674+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:23.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:22.675+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:23.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:23.190+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:23.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:23.190+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:23.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:23.190+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:23.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:23.190+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:24.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:23.771+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:24.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:23.771+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:24.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:23.771+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:24.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:23.771+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:23.639+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:23.639+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:23.639+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:23.640+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:24.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:24.221+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:24.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:24.221+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:24.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:24.221+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:24.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:24.222+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:25.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:24.732+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:25.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:24.732+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:25.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:24.732+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:25.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:24.732+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:24.662+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:24.662+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:24.662+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:24.663+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:25.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:25.271+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:25.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:25.271+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:25.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:25.271+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:25.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:25.273+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:26.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:25.683+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:26.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:25.683+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:26.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:25.683+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:26.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:25.684+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:26.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:25.713+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:25.713+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:25.713+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:25.714+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:26.523 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:54:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:54:26.401+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 38227 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:54:26.524 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:26.267+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:26.524 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:26.267+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:26.524 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:26.267+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:26.524 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:26.267+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:26.837 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:26.672+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:26.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:26.672+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:26.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:26.672+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:26.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:26.673+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:26.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:26.657+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:26.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:26.657+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:26.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:26.657+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:26.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:26.658+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:27.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:27.307+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:27.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:27.307+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:27.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:27.307+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:27.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:27.308+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:28.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:27.685+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:28.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:27.685+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:27.685+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:27.686+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:28.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:27.684+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:28.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:27.684+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:28.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:27.684+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:28.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:27.685+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:28.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:28.350+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:28.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:28.350+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:28.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:28.350+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:28.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:28.351+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:29.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:28.701+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:28.701+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:28.701+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:28.701+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:29.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:28.729+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:29.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:28.729+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:29.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:28.729+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:29.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:28.730+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:29.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:29.347+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:29.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:29.347+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:29.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:29.347+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:29.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:29.347+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:30.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:29.710+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:30.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:29.710+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:30.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:29.710+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:30.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:29.711+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:30.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:29.670+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:30.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:29.670+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:30.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:29.670+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:30.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:29.671+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:30.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:30.318+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:30.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:30.318+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:30.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:30.318+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:30.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:30.319+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:31.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:30.751+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:31.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:30.751+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:31.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:30.751+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:31.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:30.751+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:31.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:30.657+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:31.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:30.657+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:30.657+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:30.658+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:31.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:54:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:54:31.407+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 38320 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:54:31.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:31.330+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:31.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:31.330+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:31.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:31.330+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:31.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:31.331+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:32.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:31.724+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:32.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:31.724+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:32.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:31.724+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:32.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:31.725+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:31.704+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:31.704+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:31.704+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:31.705+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:32.355 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:32.291+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:32.355 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:32.291+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:32.355 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:32.291+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:32.355 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:32.292+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:33.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:32.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:33.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:32.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:33.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:32.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:33.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:32.706+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:32.685+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:32.685+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:32.685+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:33.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:32.686+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:33.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:33.337+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:33.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:33.337+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:33.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:33.337+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:33.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:33.338+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:34.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:33.641+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:33.641+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:33.641+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:33.641+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:34.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:33.710+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:34.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:33.710+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:34.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:33.710+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:34.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:33.710+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:34.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:34.326+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:34.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:34.326+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:34.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:34.326+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:34.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:34.327+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:35.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:34.670+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:34.670+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:34.670+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:34.671+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:35.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:34.742+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:35.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:34.742+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:35.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:34.742+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:35.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:34.743+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:35.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:35.349+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:35.634 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:35.349+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:35.634 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:35.349+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:35.634 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:35.349+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:36.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:35.634+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:36.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:35.634+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:36.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:35.634+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:36.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:35.635+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:36.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:35.736+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:36.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:35.736+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:36.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:35.736+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:36.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:35.737+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:36.648 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:54:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:54:36.412+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 38410 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:54:36.649 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:36.393+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:36.649 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:36.393+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:36.649 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:36.393+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:36.649 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:36.394+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:36.649+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:36.649+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:36.649+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:36.650+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:37.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:36.763+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:37.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:36.763+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:37.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:36.763+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:37.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:36.764+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:37.780 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:37.411+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:37.780 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:37.411+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:37.780 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:37.411+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:37.781 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:37.412+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:37.781 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:37.650+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:37.781 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:37.650+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:37.781 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:37.650+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:37.781 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:37.651+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:38.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:37.781+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:38.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:37.781+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:38.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:37.781+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:38.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:37.782+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:38.680 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:38.393+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:38.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:38.393+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:38.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:38.393+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:38.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:38.394+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:39.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:38.758+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:39.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:38.758+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:39.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:38.758+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:39.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:38.759+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:39.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:38.681+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:38.681+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:38.681+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:38.682+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:39.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:39.354+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:39.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:39.355+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:39.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:39.355+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:39.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:39.356+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:40.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:39.768+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:40.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:39.768+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:40.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:39.768+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:40.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:39.769+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:40.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:39.646+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:39.646+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:39.646+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:40.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:39.647+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:40.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:40.313+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:40.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:40.313+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:40.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:40.313+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:40.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:40.313+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:41.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:40.669+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:41.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:40.669+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:41.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:40.669+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:41.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:40.670+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:41.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:40.802+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:41.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:40.802+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:41.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:40.802+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:41.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:40.802+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:41.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:54:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:54:41.416+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 38502 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:54:41.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:41.324+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:41.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:41.324+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:41.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:41.324+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:41.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:41.325+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:42.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:41.818+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:42.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:41.818+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:42.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:41.818+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:42.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:41.819+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:42.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:41.699+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:42.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:41.699+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:42.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:41.699+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:42.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:41.700+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:42.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:42.359+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:42.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:42.359+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:42.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:42.359+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:42.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:42.359+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:43.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:42.681+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:43.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:42.681+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:43.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:42.681+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:43.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:42.681+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:43.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:42.787+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:43.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:42.787+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:43.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:42.787+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:43.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:42.788+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:43.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:43.332+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:43.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:43.332+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:43.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:43.332+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:43.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:43.333+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:44.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:43.709+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:44.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:43.709+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:44.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:43.709+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:44.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:43.709+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:44.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:43.807+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:44.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:43.807+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:44.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:43.807+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:44.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:43.808+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:44.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:44.321+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:44.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:44.321+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:44.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:44.321+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:44.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:44.322+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:44.818+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:44.818+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:44.818+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:44.819+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:44.719+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:44.719+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:44.719+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:44.720+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:45.762 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:45.351+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:45.763 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:45.351+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:45.763 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:45.351+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:45.763 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:45.352+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:46.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:45.866+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:46.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:45.866+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:46.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:45.866+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:46.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:45.866+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:46.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:45.762+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:46.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:45.762+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:46.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:45.762+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:46.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:45.763+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:46.806 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:54:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:54:46.421+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 38592 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:54:46.806 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:46.388+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:46.807 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:46.388+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:46.807 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:46.388+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:46.807 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:46.389+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:47.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:46.806+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:47.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:46.806+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:47.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:46.806+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:47.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:46.806+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:47.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:46.852+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:47.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:46.852+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:47.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:46.853+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:47.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:46.854+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:47.836 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:47.348+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:47.836 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:47.348+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:47.836 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:47.348+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:47.836 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:47.349+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:48.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:47.835+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:48.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:47.835+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:48.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:47.835+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:48.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:47.836+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:48.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:47.874+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:48.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:47.874+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:48.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:47.874+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:48.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:47.875+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:48.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:48.326+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:48.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:48.326+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:48.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:48.326+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:48.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:48.327+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:49.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:48.885+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:49.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:48.885+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:49.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:48.885+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:49.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:48.886+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:49.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:48.846+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:49.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:48.846+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:49.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:48.846+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:49.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:48.847+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:49.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:49.317+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:49.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:49.317+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:49.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:49.317+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:49.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:49.318+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:49.853+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:49.853+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:49.853+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:49.853+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:50.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:49.836+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:49.836+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:49.836+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:49.836+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:50.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:50.297+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:50.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:50.297+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:50.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:50.297+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:50.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:50.298+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:50.793+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:50.793+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:50.793+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:50.794+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:50.845+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:50.845+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:50.845+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:50.845+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:51.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:54:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:54:51.426+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 38685 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:54:51.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:51.330+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:51.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:51.330+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:51.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:51.330+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:51.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:51.331+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:52.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:51.787+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:52.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:51.787+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:52.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:51.787+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:51.787+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:51.892+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:51.892+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:51.892+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:51.893+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:52.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:52.335+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:52.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:52.335+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:52.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:52.335+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:52.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:52.336+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:53.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:52.795+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:53.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:52.795+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:53.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:52.795+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:53.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:52.796+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:52.905+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:52.905+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:52.905+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:52.906+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:53.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:53.291+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:53.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:53.291+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:53.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:53.291+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:53.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:53.292+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:54.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:53.785+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:54.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:53.785+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:54.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:53.785+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:54.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:53.785+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:54.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:53.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:54.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:53.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:54.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:53.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:54.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:53.861+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:54.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:54.310+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:54.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:54.310+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:54.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:54.310+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:54.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:54.311+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:55.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:54.743+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:55.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:54.743+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:55.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:54.743+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:55.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:54.744+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:55.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:54.881+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:55.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:54.881+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:55.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:54.881+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:55.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:54.881+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:55.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:55.268+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:55.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:55.268+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:55.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:55.268+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:55.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:55.269+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:56.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:55.783+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:56.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:55.783+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:56.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:55.783+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:56.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:55.784+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:56.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:55.841+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:56.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:55.841+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:56.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:55.841+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:56.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:55.842+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:56.553 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:54:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:54:56.431+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 38775 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:54:56.554 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:56.242+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:56.554 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:56.242+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:56.554 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:56.242+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:56.554 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:56.243+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:57.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:56.804+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:57.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:56.805+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:57.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:56.805+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:57.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:56.805+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:56.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:56.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:57.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:56.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:57.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:56.861+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:57.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:57.214+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:57.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:57.214+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:57.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:57.214+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:57.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:57.215+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:58.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:57.757+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:58.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:57.757+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:58.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:57.757+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:58.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:57.758+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:58.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:57.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:58.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:57.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:57.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:57.859+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:58.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:58.227+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:58.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:58.227+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:58.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:58.227+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:58.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:58.228+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:54:59.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:58.830+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:59.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:58.830+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:59.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:58.830+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:54:59.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:58.830+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:59.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:58.717+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:59.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:58.717+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:59.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:58.717+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:54:59.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:58.718+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:54:59.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:59.260+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:59.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:59.260+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:59.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:59.260+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:54:59.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:54:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:54:59.260+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:00.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:59.805+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:00.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:59.805+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:00.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:59.805+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:00.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:54:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:54:59.805+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:59.673+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:59.673+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:59.673+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:54:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:54:59.674+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:00.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:00.215+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:00.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:00.215+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:00.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:00.215+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:00.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:00.216+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:01.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:00.632+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:01.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:00.632+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:01.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:00.632+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:01.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:00.633+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:00.757+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:00.757+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:00.757+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:00.758+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:01.436 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:01.178+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:01.436 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:01.179+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:01.437 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:01.179+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:01.437 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:01.180+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:01.780 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:55:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:55:01.436+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 38867 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:55:01.780 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:01.635+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:01.781 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:01.635+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:01.781 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:01.635+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:01.781 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:01.635+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:01.807+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:01.807+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:01.807+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:01.808+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:02.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:02.178+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:02.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:02.178+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:02.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:02.178+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:02.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:02.179+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:03.075 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:02.838+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:03.075 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:02.838+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:03.075 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:02.838+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:03.075 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:02.839+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:03.076 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:02.611+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:03.076 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:02.611+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:03.076 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:02.611+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:03.076 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:02.612+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:03.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:03.131+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:03.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:03.131+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:03.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:03.131+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:03.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:03.132+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:03.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:03.584+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:03.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:03.584+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:03.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:03.584+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:03.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:03.585+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:04.138 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:03.873+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:04.138 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:03.873+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:04.138 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:03.873+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:04.138 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:03.873+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:04.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:04.137+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:04.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:04.137+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:04.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:04.137+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:04.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:04.139+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:04.875 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:04.613+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:04.875 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:04.613+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:04.875 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:04.613+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:04.876 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:04.614+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:05.137 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:04.875+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:05.137 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:04.875+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:05.137 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:04.875+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:05.137 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:04.875+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:05.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:05.137+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:05.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:05.137+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:05.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:05.137+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:05.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:05.138+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:06.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:05.831+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:06.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:05.831+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:06.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:05.831+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:06.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:05.832+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:05.608+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:05.608+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:05.608+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:05.609+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:06.441 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:06.098+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:06.441 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:06.098+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:06.441 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:06.098+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:06.442 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:06.099+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:06.837 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:55:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:55:06.441+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 38957 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:55:06.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:06.623+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:06.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:06.623+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:06.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:06.623+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:06.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:06.624+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:07.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:07.054+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:07.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:07.054+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:07.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:07.054+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:07.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:07.055+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:07.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:06.868+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:07.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:06.868+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:07.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:06.868+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:07.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:06.869+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:07.908 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:07.604+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:07.908 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:07.604+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:07.908 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:07.604+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:07.908 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:07.605+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:08.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:08.095+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:08.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:08.095+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:08.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:08.095+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:08.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:08.096+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:07.908+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:07.908+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:08.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:07.908+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:08.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:07.909+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:08.939 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:08.591+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:08.940 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:08.591+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:08.940 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:08.591+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:08.940 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:08.592+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:09.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:08.939+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:09.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:08.939+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:09.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:08.939+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:09.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:08.940+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:09.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:09.121+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:09.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:09.121+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:09.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:09.121+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:09.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:09.121+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:09.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:09.578+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:09.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:09.578+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:09.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:09.578+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:09.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:09.578+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:10.240 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:10.083+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:10.241 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:10.083+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:10.241 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:10.083+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:10.241 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:10.084+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:10.241 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:09.969+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:10.241 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:09.969+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:10.241 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:09.969+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:10.242 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:09.970+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:10.929 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:10.596+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:10.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:10.596+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:10.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:10.596+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:10.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:10.597+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:11.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:10.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:11.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:10.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:11.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:10.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:11.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:10.930+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:11.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:11.069+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:11.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:11.069+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:11.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:11.069+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:11.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:11.070+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:11.781 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:55:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:55:11.445+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 39050 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:55:11.782 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:11.596+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:11.782 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:11.596+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:11.782 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:11.596+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:11.782 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:11.596+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:12.059 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:11.890+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:12.059 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:11.890+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:12.059 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:11.890+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:12.059 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:11.891+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:12.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:12.058+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:12.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:12.058+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:12.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:12.058+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:12.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:12.059+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:12.906 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:12.618+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:12.906 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:12.618+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:12.906 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:12.618+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:12.906 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:12.618+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:12.907 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:12.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:12.907 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:12.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:12.907 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:12.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:13.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:12.906+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:13.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:13.084+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:13.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:13.084+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:13.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:13.084+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:13.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:13.085+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:14.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:13.653+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:14.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:13.653+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:14.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:13.653+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:14.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:13.655+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:14.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:13.876+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:13.876+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:13.876+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:13.877+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:14.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:14.095+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:14.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:14.095+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:14.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:14.095+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:14.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:14.096+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:15.055 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:14.656+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:15.056 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:14.656+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:15.056 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:14.656+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:15.056 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:14.656+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:15.056 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:14.903+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:15.056 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:14.903+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:15.056 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:14.903+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:15.056 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:14.904+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:15.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:15.055+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:15.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:15.055+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:15.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:15.055+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:15.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:15.056+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:16.080 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:15.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:16.080 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:15.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:16.081 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:15.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:16.081 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:15.705+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:16.081 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:15.913+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:16.081 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:15.913+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:16.082 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:15.913+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:16.082 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:15.914+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:16.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:16.080+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:16.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:16.080+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:16.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:16.080+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:16.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:16.081+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:16.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:55:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:55:16.450+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 39140 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:55:16.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:16.667+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:16.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:16.667+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:16.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:16.667+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:16.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:16.668+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:17.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:17.080+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:17.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:17.080+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:17.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:17.080+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:17.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:17.081+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:17.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:16.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:17.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:16.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:17.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:16.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:17.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:16.931+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:17.979 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:17.640+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:17.979 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:17.640+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:17.979 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:17.640+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:17.979 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:17.640+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:18.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:18.083+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:18.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:18.083+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:18.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:18.083+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:18.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:18.084+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:18.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:17.978+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:18.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:17.978+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:18.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:17.978+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:18.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:17.979+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:18.960 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:18.686+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:18.960 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:18.686+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:18.960 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:18.686+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:18.960 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:18.687+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:19.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:19.094+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:19.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:19.094+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:19.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:19.094+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:19.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:19.095+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:19.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:18.960+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:19.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:18.960+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:19.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:18.960+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:19.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:18.961+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:20.010 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:19.698+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:20.011 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:19.698+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:20.011 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:19.698+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:20.011 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:19.699+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:20.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:20.045+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:20.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:20.045+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:20.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:20.045+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:20.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:20.046+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:20.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:20.010+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:20.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:20.010+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:20.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:20.010+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:20.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:20.011+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:21.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:20.693+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:21.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:20.693+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:21.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:20.693+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:21.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:20.694+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:21.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:21.020+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:21.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:21.020+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:21.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:21.020+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:21.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:21.021+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:21.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:21.042+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:21.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:21.042+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:21.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:21.042+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:21.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:21.043+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:21.781 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:55:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:55:21.455+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 39232 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:55:21.781 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:21.651+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:21.782 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:21.651+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:21.782 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:21.651+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:21.782 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:21.652+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:22.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:22.065+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:22.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:22.065+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:22.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:22.065+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:22.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:22.066+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:22.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:22.046+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:22.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:22.046+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:22.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:22.046+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:22.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:22.047+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:23.034 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:22.664+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:23.034 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:22.664+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:23.034 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:22.664+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:23.035 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:22.665+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:23.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:23.034+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:23.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:23.034+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:23.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:23.034+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:23.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:23.035+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:23.036+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:23.036+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:23.036+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:23.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:23.037+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:24.013 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:23.670+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:24.013 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:23.670+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:24.013 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:23.670+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:24.013 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:23.671+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:24.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:24.073+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:24.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:24.073+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:24.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:24.073+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:24.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:24.074+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:24.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:24.013+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:24.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:24.013+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:24.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:24.013+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:24.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:24.014+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:24.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:24.696+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:24.696+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:24.696+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:24.697+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:25.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:25.061+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:25.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:25.061+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:25.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:25.061+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:25.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:25.062+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:25.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:25.034+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:25.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:25.034+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:25.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:25.034+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:25.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:25.035+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:25.995 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:25.697+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:25.995 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:25.697+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:25.995 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:25.697+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:25.995 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:25.697+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:26.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:25.995+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:26.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:25.995+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:26.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:25.995+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:26.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:25.996+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:26.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:26.048+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:26.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:26.048+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:26.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:26.048+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:26.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:26.048+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:26.712 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:55:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:55:26.460+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 39322 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:55:26.984 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:26.712+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:26.984 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:26.712+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:26.985 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:26.712+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:26.985 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:26.713+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:27.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:26.984+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:27.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:26.984+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:27.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:26.984+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:27.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:26.985+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:27.094+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:27.094+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:27.094+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:27.095+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:27.995 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:27.744+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:27.996 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:27.744+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:27.996 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:27.744+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:27.996 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:27.744+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:28.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:27.995+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:28.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:27.995+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:28.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:27.995+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:28.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:27.996+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:28.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:28.056+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:28.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:28.056+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:28.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:28.056+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:28.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:28.056+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:29.024 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:28.758+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:29.025 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:28.758+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:29.025 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:28.758+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:29.025 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:28.759+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:29.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:29.024+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:29.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:29.024+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:29.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:29.024+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:29.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:29.025+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:29.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:29.095+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:29.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:29.095+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:29.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:29.095+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:29.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:29.096+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:30.029 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:29.712+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:30.029 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:29.712+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:30.030 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:29.712+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:30.030 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:29.713+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:30.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:30.085+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:30.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:30.085+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:30.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:30.085+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:30.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:30.085+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:30.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:30.029+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:30.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:30.029+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:30.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:30.029+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:30.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:30.030+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:31.067 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:30.747+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:31.067 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:30.747+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:31.067 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:30.747+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:31.067 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:30.747+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:31.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:31.072+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:31.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:31.072+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:31.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:31.072+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:31.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:31.073+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:31.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:31.067+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:31.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:31.067+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:31.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:31.067+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:31.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:31.067+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:31.782 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:55:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:55:31.465+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 39415 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:55:31.782 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:31.704+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:31.782 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:31.704+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:31.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:31.704+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:31.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:31.705+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:32.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:32.029+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:32.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:32.029+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:32.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:32.029+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:32.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:32.030+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:32.024+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:32.024+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:32.024+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:32.025+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:33.010 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:32.701+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:33.011 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:32.701+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:33.011 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:32.701+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:33.011 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:32.702+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:33.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:33.023+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:33.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:33.023+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:33.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:33.023+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:33.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:33.024+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:33.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:33.010+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:33.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:33.010+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:33.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:33.010+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:33.011+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:33.967 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:33.703+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:33.967 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:33.703+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:33.967 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:33.703+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:33.968 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:33.703+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:34.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:33.967+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:34.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:33.967+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:34.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:33.967+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:34.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:33.968+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:34.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:34.056+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:34.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:34.056+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:34.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:34.056+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:34.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:34.057+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:34.936 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:34.673+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:34.937 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:34.674+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:34.937 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:34.674+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:34.937 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:34.674+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:35.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:34.936+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:35.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:34.936+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:35.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:34.936+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:35.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:34.937+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:35.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:35.019+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:35.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:35.019+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:35.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:35.019+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:35.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:35.020+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:35.918 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:35.654+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:35.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:35.654+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:35.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:35.654+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:35.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:35.655+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:36.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:36.053+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:36.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:36.053+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:36.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:36.053+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:36.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:36.055+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:36.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:35.918+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:36.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:35.918+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:36.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:35.918+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:36.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:35.919+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:36.837 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:55:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:55:36.470+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 39505 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:55:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:36.618+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:36.618+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:36.618+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:36.618+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:37.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:37.056+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:37.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:37.056+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:37.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:37.056+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:37.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:37.057+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:37.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:36.958+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:37.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:36.958+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:37.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:36.958+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:37.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:36.958+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:37.992 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:37.608+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:37.993 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:37.608+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:37.993 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:37.608+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:37.993 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:37.608+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:38.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:37.992+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:37.992+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:37.992+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:37.993+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:38.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:38.020+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:38.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:38.020+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:38.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:38.020+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:38.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:38.021+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:39.010 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:38.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:39.010 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:38.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:39.010 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:38.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:39.010 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:38.643+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:39.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:39.010+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:39.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:39.010+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:39.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:39.010+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:39.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:39.010+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:39.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:39.012+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:39.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:39.012+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:39.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:39.012+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:39.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:39.013+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:40.020 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:39.597+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:40.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:39.597+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:40.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:39.597+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:40.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:39.597+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:40.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:40.022+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:40.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:40.022+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:40.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:40.022+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:40.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:40.022+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:40.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:40.020+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:40.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:40.020+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:40.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:40.020+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:40.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:40.021+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:40.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:40.550+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:40.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:40.550+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:40.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:40.550+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:40.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:40.551+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:41.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:40.985+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:41.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:40.985+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:41.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:40.985+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:41.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:40.986+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:41.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:41.064+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:41.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:41.064+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:41.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:41.064+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:41.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:41.065+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:41.782 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:55:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:55:41.474+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 39597 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:55:41.782 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:41.552+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:41.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:41.552+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:41.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:41.552+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:41.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:41.553+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:42.032 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:42.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:42.032 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:42.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:42.032 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:42.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:42.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:42.033+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:42.106+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:42.106+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:42.106+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:42.106+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:43.011 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:42.603+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:43.011 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:42.603+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:43.011 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:42.603+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:43.011 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:42.603+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:43.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:43.140+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:43.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:43.140+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:43.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:43.140+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:43.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:43.141+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:43.082+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:43.082+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:43.082+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:43.083+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:44.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:43.608+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:44.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:43.608+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:44.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:43.608+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:44.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:43.609+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:44.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:44.116+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:44.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:44.116+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:44.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:44.116+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:44.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:44.117+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:44.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:44.139+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:44.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:44.139+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:44.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:44.139+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:44.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:44.140+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:45.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:44.641+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:44.641+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:44.641+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:44.642+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:45.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:45.130+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:45.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:45.130+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:45.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:45.130+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:45.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:45.131+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:45.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:45.119+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:45.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:45.119+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:45.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:45.119+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:45.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:45.119+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:46.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:45.671+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:46.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:45.671+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:46.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:45.671+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:46.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:45.672+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:46.480 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:46.156+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:46.480 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:46.156+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:46.480 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:46.156+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:46.480 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:46.157+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:46.480 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:46.099+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:46.481 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:46.099+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:46.481 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:46.099+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:46.481 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:46.100+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:46.837 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:55:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:55:46.479+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 39687 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:55:46.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:46.711+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:46.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:46.711+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:46.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:46.711+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:46.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:46.712+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:47.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:47.109+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:47.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:47.109+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:47.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:47.109+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:47.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:47.110+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:47.108+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:47.108+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:47.108+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:47.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:47.109+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:48.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:47.704+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:48.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:47.704+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:48.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:47.704+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:48.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:47.705+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:48.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:48.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:48.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:48.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:48.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:48.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:48.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:48.132+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:48.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:48.125+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:48.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:48.125+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:48.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:48.125+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:48.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:48.125+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:49.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:48.707+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:49.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:48.707+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:49.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:48.707+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:49.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:48.708+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:49.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:49.118+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:49.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:49.118+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:49.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:49.118+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:49.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:49.119+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:49.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:49.092+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:49.092+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:49.092+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:49.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:49.093+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:50.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:49.662+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:50.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:49.662+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:50.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:49.662+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:50.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:49.662+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:50.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:50.111+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:50.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:50.111+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:50.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:50.111+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:50.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:50.112+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:50.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:50.138+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:50.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:50.138+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:50.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:50.138+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:50.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:50.138+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:51.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:50.669+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:50.669+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:50.669+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:50.670+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:51.484 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:51.128+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:51.484 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:51.128+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:51.484 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:51.128+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:51.485 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:51.128+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:51.485 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:51.182+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:51.485 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:51.182+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:51.485 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:51.182+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:51.485 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:51.182+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:51.783 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:55:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:55:51.484+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 39780 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:55:51.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:51.634+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:51.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:51.634+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:51.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:51.634+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:51.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:51.635+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:52.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:52.123+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:52.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:52.123+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:52.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:52.123+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:52.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:52.124+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:52.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:52.196+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:52.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:52.196+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:52.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:52.196+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:52.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:52.196+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:53.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:52.593+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:53.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:52.593+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:53.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:52.593+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:53.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:52.594+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:53.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:53.125+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:53.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:53.125+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:53.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:53.125+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:53.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:53.126+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:53.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:53.237+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:53.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:53.237+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:53.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:53.237+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:53.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:53.238+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:54.080 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:53.630+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:54.080 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:53.630+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:54.081 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:53.630+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:54.081 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:53.631+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:54.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:54.081+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:54.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:54.081+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:54.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:54.081+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:54.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:54.082+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:54.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:54.219+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:54.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:54.220+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:54.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:54.220+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:54.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:54.220+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:55.048 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:54.614+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:55.048 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:54.614+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:55.048 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:54.614+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:55.048 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:54.615+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:55.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:55.049+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:55.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:55.049+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:55.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:55.049+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:55.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:55.049+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:55.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:55.180+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:55.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:55.180+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:55.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:55.180+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:55.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:55.181+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:56.043 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:55.597+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:56.043 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:55.597+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:56.044 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:55.597+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:56.044 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:55.598+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:56.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:56.044+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:56.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:56.044+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:56.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:56.044+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:56.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:56.045+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:56.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:56.209+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:56.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:56.209+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:56.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:56.209+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:56.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:56.209+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:56.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:55:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:55:56.490+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 39870 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:55:56.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:56.552+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:56.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:56.552+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:56.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:56.552+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:56.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:56.552+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:57.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:57.037+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:57.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:57.037+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:57.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:57.037+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:57.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:57.039+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:57.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:57.193+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:57.193+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:57.193+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:57.194+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:57.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:57.586+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:57.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:57.586+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:57.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:57.586+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:57.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:57.587+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:58.189 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:58.053+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:58.189 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:58.053+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:58.189 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:58.053+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:58.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:58.054+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:58.568 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:58.190+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:58.568 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:58.190+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:58.568 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:58.190+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:58.568 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:58.191+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:58.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:58.569+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:58.569+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:58.569+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:55:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:58.570+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:55:59.214 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:59.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:59.214 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:59.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:59.214 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:59.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:55:59.214 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:55:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:55:59.022+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:55:59.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:59.215+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:59.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:59.215+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:59.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:59.215+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:55:59.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:55:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:55:59.215+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:00.014 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:59.613+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:00.014 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:59.613+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:00.014 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:59.613+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:00.014 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:55:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:55:59.614+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:00.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:00.013+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:00.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:00.013+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:00.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:00.013+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:00.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:00.014+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:00.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:00.230+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:00.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:00.230+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:00.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:00.230+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:00.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:00.231+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:01.015 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:00.594+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:01.016 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:00.594+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:01.016 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:00.594+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:01.016 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:00.595+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:01.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:01.015+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:01.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:01.015+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:01.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:01.015+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:01.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:01.016+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:01.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:01.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:01.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:01.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:01.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:01.240+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:01.783 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:56:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:56:01.494+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 39962 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:56:01.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:01.632+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:01.784 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:01.632+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:01.784 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:01.632+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:01.784 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:01.633+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:02.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:02.036+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:02.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:02.036+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:02.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:02.036+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:02.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:02.037+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:02.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:02.251+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:02.251+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:02.251+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:02.252+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:03.005 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:02.605+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:03.005 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:02.605+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:03.005 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:02.605+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:03.005 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:02.606+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:03.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:03.005+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:03.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:03.005+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:03.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:03.005+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:03.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:03.006+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:03.214+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:03.214+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:03.214+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:03.215+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:03.963 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:03.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:03.963 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:03.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:03.963 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:03.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:03.963 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:03.588+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:04.232 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:03.963+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:04.232 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:03.963+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:04.232 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:03.963+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:04.233 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:03.964+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:04.515 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:04.232+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:04.515 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:04.232+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:04.515 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:04.232+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:04.515 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:04.233+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:04.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:04.546+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:04.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:04.546+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:04.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:04.546+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:04.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:04.547+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:05.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:05.013+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:05.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:05.013+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:05.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:05.013+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:05.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:05.013+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:05.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:05.191+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:05.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:05.191+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:05.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:05.191+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:05.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:05.192+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:05.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:05.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:05.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:05.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:05.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:05.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:05.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:05.507+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:06.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:06.005+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:06.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:06.005+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:06.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:06.005+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:06.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:06.005+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:06.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:06.202+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:06.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:06.202+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:06.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:06.202+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:06.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:06.203+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:06.837 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:56:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:56:06.498+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 40052 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:56:06.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:06.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:06.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:06.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:06.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:06.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:06.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:06.510+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:07.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:07.223+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:07.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:07.223+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:07.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:07.223+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:07.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:07.224+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:07.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:07.049+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:07.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:07.049+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:07.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:07.049+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:07.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:07.050+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:07.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:07.493+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:07.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:07.493+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:07.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:07.493+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:07.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:07.494+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:08.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:08.264+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:08.264+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:08.264+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:08.264+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:08.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:08.039+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:08.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:08.039+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:08.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:08.039+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:08.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:08.039+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:08.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:08.447+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:08.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:08.447+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:08.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:08.447+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:08.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:08.448+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:09.315 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:09.248+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:09.315 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:09.248+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:09.315 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:09.248+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:09.315 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:09.248+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:09.316 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:09.037+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:09.316 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:09.037+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:09.316 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:09.037+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:09.316 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:09.037+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:09.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:09.477+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:09.477+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:09.477+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:09.477+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:10.274 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:10.029+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:10.274 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:10.029+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:10.275 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:10.029+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:10.275 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:10.029+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:10.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:10.505+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:10.505+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:10.505+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:10.505+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:10.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:10.274+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:10.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:10.274+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:10.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:10.274+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:10.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:10.275+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:11.310 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:10.991+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:11.311 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:10.992+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:11.311 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:10.992+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:11.311 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:10.992+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:11.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:56:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:56:11.503+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 40144 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:56:11.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:11.474+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:11.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:11.474+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:11.474+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:11.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:11.475+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:11.310+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:11.310+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:11.310+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:11.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:11.311+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:12.320 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:11.988+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:12.320 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:11.988+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:12.320 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:11.988+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:12.320 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:11.989+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:12.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:12.494+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:12.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:12.494+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:12.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:12.494+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:12.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:12.495+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:12.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:12.320+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:12.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:12.320+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:12.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:12.320+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:12.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:12.320+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:13.320 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:12.999+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:13.320 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:12.999+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:13.321 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:12.999+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:13.321 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:12.999+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:13.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:13.475+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:13.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:13.475+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:13.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:13.475+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:13.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:13.476+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:13.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:13.320+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:13.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:13.320+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:13.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:13.320+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:13.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:13.321+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:14.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:13.978+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:13.978+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:13.978+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:13.979+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:14.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:14.336+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:14.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:14.336+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:14.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:14.336+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:14.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:14.336+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:14.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:14.428+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:14.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:14.428+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:14.428+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:14.429+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:15.305 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:15.028+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:15.305 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:15.028+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:15.305 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:15.028+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:15.306 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:15.029+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:15.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:15.387+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:15.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:15.387+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:15.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:15.387+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:15.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:15.387+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:15.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:15.305+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:15.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:15.305+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:15.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:15.305+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:15.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:15.306+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:16.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:16.053+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:16.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:16.053+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:16.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:16.053+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:16.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:16.054+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:16.631 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:56:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:56:16.508+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 40234 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:56:16.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:16.414+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:16.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:16.414+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:16.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:16.414+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:16.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:16.415+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:16.632 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:16.340+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:16.632 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:16.340+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:16.632 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:16.340+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:16.633 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:16.341+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:17.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:17.064+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:17.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:17.064+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:17.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:17.064+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:17.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:17.065+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:17.292+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:17.292+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:17.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:17.292+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:17.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:17.292+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:17.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:17.397+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:17.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:17.397+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:17.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:17.397+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:17.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:17.398+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:18.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:18.253+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:18.253+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:18.253+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:18.254+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:18.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:18.028+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:18.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:18.028+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:18.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:18.028+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:18.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:18.029+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:18.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:18.388+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:18.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:18.388+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:18.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:18.388+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:18.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:18.388+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:19.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:18.994+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:19.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:18.994+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:19.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:18.994+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:19.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:18.995+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:19.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:19.205+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:19.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:19.205+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:19.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:19.205+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:19.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:19.206+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:19.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:19.350+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:19.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:19.350+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:19.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:19.350+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:19.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:19.350+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:20.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:20.171+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:20.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:20.171+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:20.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:20.171+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:20.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:20.172+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:20.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:20.002+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:20.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:20.002+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:20.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:20.002+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:20.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:20.002+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:20.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:20.397+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:20.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:20.397+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:20.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:20.397+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:20.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:20.398+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:21.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:20.993+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:21.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:20.993+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:21.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:20.993+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:21.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:20.994+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:21.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:21.192+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:21.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:21.192+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:21.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:21.192+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:21.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:21.192+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:21.784 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:56:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:56:21.513+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 40326 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:56:21.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:21.389+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:21.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:21.389+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:21.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:21.389+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:21.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:21.389+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:22.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:21.988+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:21.988+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:21.988+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:21.989+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:22.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:22.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:22.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:22.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:22.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:22.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:22.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:22.362+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:22.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:22.229+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:22.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:22.229+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:22.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:22.229+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:22.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:22.230+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:23.204 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:22.984+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:23.205 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:22.984+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:23.205 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:22.984+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:23.205 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:22.985+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:23.403 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:23.204+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:23.404 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:23.204+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:23.404 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:23.204+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:23.404 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:23.205+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:23.404 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:23.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:23.404 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:23.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:23.405 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:23.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:23.405 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:23.402+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:24.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:24.188+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:24.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:24.188+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:24.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:24.188+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:24.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:24.188+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:24.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:23.992+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:24.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:23.992+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:24.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:23.992+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:24.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:23.993+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:24.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:24.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:24.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:24.401+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:25.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:24.994+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:25.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:24.994+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:25.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:24.994+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:25.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:24.994+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:25.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:25.168+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:25.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:25.168+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:25.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:25.168+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:25.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:25.168+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:25.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:25.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:25.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:25.401+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:26.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:26.187+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:26.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:26.187+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:26.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:26.187+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:26.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:26.188+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:26.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:25.982+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:26.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:25.982+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:26.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:25.982+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:26.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:25.983+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:26.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:56:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:56:26.518+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 40415 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:56:26.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:26.434+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:26.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:26.434+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:26.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:26.434+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:26.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:26.435+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:27.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:26.952+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:27.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:26.952+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:27.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:26.952+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:27.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:26.953+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:27.191+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:27.191+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:27.191+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:27.192+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:27.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:27.484+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:27.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:27.484+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:27.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:27.484+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:27.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:27.484+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:28.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:28.155+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:28.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:28.155+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:28.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:28.155+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:28.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:28.156+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:28.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:27.921+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:28.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:27.921+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:28.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:27.921+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:28.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:27.922+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:28.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:28.512+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:28.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:28.512+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:28.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:28.512+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:28.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:28.513+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:29.189 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:28.927+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:29.189 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:28.927+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:29.189 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:28.927+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:29.189 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:28.928+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:29.492 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:29.188+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:29.492 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:29.188+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:29.492 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:29.188+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:29.493 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:29.189+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:29.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:29.492+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:29.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:29.492+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:29.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:29.492+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:29.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:29.493+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:30.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:29.925+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:30.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:29.925+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:30.216 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:29.925+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:30.216 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:29.925+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:30.493 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:30.215+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:30.493 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:30.215+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:30.493 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:30.215+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:30.493 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:30.216+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:30.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:30.493+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:30.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:30.493+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:30.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:30.493+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:30.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:30.494+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:31.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:30.944+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:31.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:30.944+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:31.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:30.944+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:31.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:30.945+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:31.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:31.181+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:31.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:31.181+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:31.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:31.181+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:31.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:31.181+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:31.785 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:56:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:56:31.523+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 40508 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:56:31.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:31.528+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:31.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:31.528+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:31.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:31.528+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:31.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:31.530+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:32.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:31.949+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:32.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:31.949+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:32.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:31.949+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:32.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:31.949+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:32.508 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:32.168+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:32.508 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:32.168+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:32.508 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:32.168+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:32.509 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:32.169+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:32.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:32.508+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:32.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:32.508+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:32.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:32.508+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:32.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:32.509+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:33.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:32.942+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:33.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:32.942+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:33.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:32.942+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:33.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:32.943+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:33.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:33.135+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:33.135+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:33.135+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:33.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:33.135+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:33.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:33.489+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:33.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:33.489+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:33.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:33.489+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:33.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:33.490+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:34.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:33.991+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:34.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:33.991+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:34.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:33.991+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:34.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:33.992+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:34.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:34.135+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:34.135+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:34.135+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:34.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:34.136+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:34.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:34.513+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:34.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:34.513+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:34.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:34.513+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:34.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:34.514+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:35.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:35.007+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:35.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:35.007+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:35.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:35.007+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:35.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:35.008+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:35.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:35.108+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:35.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:35.108+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:35.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:35.108+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:35.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:35.109+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:35.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:35.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:35.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:35.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:35.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:35.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:35.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:35.510+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:36.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:36.145+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:36.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:36.145+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:36.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:36.145+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:36.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:36.145+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:36.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:35.962+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:36.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:35.962+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:36.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:35.962+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:36.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:35.962+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:36.837 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:56:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:56:36.528+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 40598 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:56:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:36.517+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:36.517+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:36.517+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:36.517+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:37.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:37.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:37.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:37.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:37.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:37.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:37.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:37.163+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:37.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:36.925+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:37.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:36.925+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:37.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:36.925+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:37.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:36.926+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:37.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:37.556+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:37.556+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:37.556+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:37.557+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:38.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:38.128+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:38.128+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:38.128+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:38.129+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:38.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:37.932+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:38.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:37.932+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:38.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:37.932+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:38.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:37.933+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:38.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:38.539+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:38.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:38.539+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:38.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:38.539+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:38.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:38.539+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:39.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:39.179+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:39.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:39.179+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:39.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:39.179+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:39.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:39.179+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:39.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:38.965+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:39.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:38.965+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:39.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:38.965+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:39.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:38.966+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:39.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:39.568+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:39.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:39.568+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:39.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:39.568+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:39.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:39.568+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:40.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:39.923+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:40.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:39.923+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:40.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:39.923+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:40.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:39.924+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:40.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:40.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:40.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:40.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:40.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:40.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:40.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:40.142+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:40.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:40.562+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:40.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:40.562+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:40.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:40.562+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:40.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:40.563+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:41.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:40.943+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:41.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:40.943+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:41.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:40.943+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:41.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:40.944+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:41.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:41.139+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:41.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:41.139+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:41.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:41.139+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:41.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:41.140+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:41.657 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:56:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:56:41.533+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 40689 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:56:41.657 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:41.518+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:41.657 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:41.518+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:41.657 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:41.518+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:41.657 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:41.519+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:42.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:42.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:42.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:42.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:42.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:42.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:42.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:42.138+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:42.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:41.974+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:42.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:41.974+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:42.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:41.974+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:42.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:41.975+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:42.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:42.471+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:42.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:42.471+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:42.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:42.471+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:42.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:42.471+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:43.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:42.928+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:43.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:42.928+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:43.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:42.928+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:43.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:42.929+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:43.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:43.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:43.137+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:43.137+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:43.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:43.511+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:43.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:43.511+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:43.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:43.511+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:43.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:43.511+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:43.885+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:43.885+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:43.885+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:43.886+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:44.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:44.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:44.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:44.132+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:44.499+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:44.499+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:44.499+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:44.500+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:45.146 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:44.871+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:45.146 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:44.871+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:45.146 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:44.871+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:45.146 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:44.871+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:45.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:45.145+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:45.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:45.145+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:45.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:45.145+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:45.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:45.147+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:45.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:45.451+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:45.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:45.451+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:45.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:45.451+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:45.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:45.452+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:46.175 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:45.864+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:46.175 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:45.864+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:46.176 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:45.864+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:46.176 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:45.865+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:46.483 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:46.175+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:46.484 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:46.175+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:46.484 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:46.175+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:46.484 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:46.176+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:46.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:56:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:56:46.539+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 40779 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:56:46.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:46.483+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:46.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:46.483+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:46.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:46.483+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:46.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:46.484+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:47.127 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:46.877+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:47.127 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:46.877+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:47.127 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:46.877+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:47.128 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:46.878+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:47.128 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:47.126+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:47.128 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:47.126+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:47.128 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:47.126+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:47.128 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:47.127+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:47.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:47.528+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:47.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:47.528+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:47.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:47.528+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:47.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:47.529+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:48.136 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:47.850+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:48.137 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:47.850+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:48.137 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:47.850+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:48.137 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:47.850+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:48.496 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:48.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:48.497 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:48.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:48.497 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:48.136+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:48.497 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:48.137+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:48.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:48.496+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:48.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:48.496+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:48.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:48.496+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:48.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:48.497+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:49.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:48.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:49.140 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:48.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:49.140 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:48.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:49.140 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:48.868+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:49.463 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:49.139+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:49.463 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:49.139+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:49.463 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:49.139+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:49.463 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:49.140+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:49.820 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:49.463+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:49.820 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:49.463+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:49.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:49.463+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:49.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:49.464+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:50.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:49.820+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:49.820+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:49.820+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:49.821+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:50.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:50.101+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:50.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:50.101+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:50.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:50.101+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:50.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:50.102+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:50.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:50.473+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:50.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:50.473+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:50.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:50.473+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:50.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:50.474+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:51.121 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:50.865+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:51.122 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:50.865+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:51.122 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:50.865+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:51.122 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:50.866+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:51.453 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:51.121+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:51.454 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:51.121+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:51.454 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:51.121+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:51.454 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:51.122+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:51.785 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:56:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:56:51.544+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 40872 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:56:51.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:51.453+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:51.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:51.453+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:51.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:51.453+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:51.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:51.454+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:52.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:51.817+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:52.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:51.817+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:52.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:51.817+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:52.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:51.817+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:52.469 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:52.110+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:52.469 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:52.110+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:52.469 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:52.110+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:52.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:52.111+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:52.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:52.469+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:52.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:52.469+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:52.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:52.469+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:52.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:52.470+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:53.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:52.851+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:53.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:52.851+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:53.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:52.851+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:53.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:52.852+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:53.453 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:53.159+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:53.454 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:53.159+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:53.454 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:53.159+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:53.454 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:53.160+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:53.827 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:53.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:53.828 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:53.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:53.828 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:53.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:53.828 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:53.454+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:54.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:53.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:54.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:53.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:54.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:53.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:54.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:53.828+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:54.406 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:54.149+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:54.406 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:54.149+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:54.406 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:54.149+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:54.407 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:54.150+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:54.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:54.406+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:54.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:54.406+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:54.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:54.406+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:54.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:54.407+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:55.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:54.799+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:55.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:54.799+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:55.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:54.799+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:55.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:54.801+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:55.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:55.397+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:55.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:55.397+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:55.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:55.397+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:55.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:55.398+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:55.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:55.171+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:55.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:55.171+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:55.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:55.171+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:55.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:55.171+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:56.177 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:55.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:56.177 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:55.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:56.178 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:55.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:56.178 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:55.847+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:56.431 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:56.177+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:56.432 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:56.177+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:56.432 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:56.177+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:56.432 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:56.178+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:56.837 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:56:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:56:56.549+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 40962 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:56:56.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:56.431+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:56.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:56.431+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:56.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:56.431+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:56.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:56.433+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:57.153 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:56.876+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:57.154 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:56.876+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:57.154 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:56.876+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:57.154 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:56.877+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:57.442 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:57.153+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:57.443 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:57.153+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:57.443 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:57.153+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:57.443 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:57.154+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:57.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:57.442+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:57.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:57.442+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:57.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:57.442+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:57.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:57.443+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:58.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:57.876+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:58.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:57.876+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:58.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:57.876+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:58.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:57.877+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:58.469 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:58.202+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:58.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:58.202+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:58.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:58.202+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:58.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:58.203+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:58.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:58.469+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:58.469+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:58.469+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:58.470+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:59.236 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:58.910+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:59.236 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:58.910+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:59.236 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:58.910+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:56:59.236 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:58.911+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:56:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:59.421+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:59.422+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:59.422+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:56:59.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:56:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:56:59.423+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:56:59.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:59.236+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:59.236+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:59.236+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:56:59.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:56:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:56:59.236+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:00.239 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:59.891+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:00.239 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:59.891+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:00.239 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:59.891+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:00.240 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:56:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:56:59.891+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:00.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:00.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:00.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:00.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:00.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:00.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:00.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:00.240+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:00.407+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:00.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:00.407+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:00.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:00.407+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:00.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:00.408+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:01.242 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:00.851+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:01.242 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:00.851+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:01.243 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:00.851+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:01.243 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:00.852+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:01.554 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:01.398+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:01.555 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:01.398+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:01.555 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:01.398+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:01.555 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:01.399+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:01.555 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:01.242+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:01.555 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:01.242+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:01.555 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:01.242+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:01.556 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:01.243+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:01.837 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:57:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:57:01.554+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 41054 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:57:02.214 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:01.838+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:02.214 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:01.838+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:02.214 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:01.838+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:02.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:01.839+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:02.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:02.214+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:02.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:02.214+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:02.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:02.214+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:02.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:02.215+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:02.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:02.375+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:02.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:02.375+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:02.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:02.375+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:02.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:02.376+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:03.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:02.804+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:03.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:02.804+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:03.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:02.804+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:03.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:02.805+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:03.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:03.344+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:03.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:03.344+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:03.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:03.344+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:03.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:03.345+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:03.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:03.203+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:03.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:03.203+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:03.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:03.203+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:03.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:03.204+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:04.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:03.787+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:04.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:03.787+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:04.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:03.787+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:04.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:03.788+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:04.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:04.189+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:04.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:04.189+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:04.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:04.189+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:04.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:04.190+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:04.312+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:04.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:04.312+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:04.312+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:04.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:04.313+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:05.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:04.803+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:04.803+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:04.803+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:05.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:04.804+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:05.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:05.348+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:05.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:05.348+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:05.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:05.348+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:05.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:05.349+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:05.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:05.165+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:05.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:05.165+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:05.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:05.165+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:05.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:05.166+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:06.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:05.815+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:06.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:05.815+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:06.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:05.815+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:06.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:05.815+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:06.560 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:06.341+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:06.560 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:06.341+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:06.560 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:06.341+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:06.560 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:06.342+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:06.560 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:06.184+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:06.561 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:06.184+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:06.561 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:06.184+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:06.561 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:06.185+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:06.837 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:57:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:57:06.559+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 41144 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:57:06.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:06.795+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:06.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:06.795+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:06.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:06.795+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:06.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:06.796+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:07.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:07.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:07.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:07.362+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:07.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:07.362+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:07.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:07.152+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:07.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:07.152+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:07.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:07.152+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:07.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:07.152+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:08.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:07.796+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:08.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:07.796+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:08.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:07.796+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:08.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:07.796+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:08.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:08.330+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:08.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:08.330+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:08.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:08.330+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:08.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:08.330+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:08.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:08.180+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:08.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:08.180+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:08.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:08.180+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:08.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:08.181+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:09.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:08.750+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:09.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:08.750+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:09.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:08.750+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:09.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:08.751+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:09.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:09.323+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:09.323+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:09.323+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:09.324+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:09.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:09.173+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:09.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:09.173+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:09.173+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:09.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:09.173+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:10.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:09.767+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:10.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:09.767+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:10.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:09.767+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:10.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:09.767+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:10.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:10.200+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:10.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:10.200+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:10.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:10.200+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:10.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:10.200+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:10.282+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:10.282+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:10.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:10.282+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:10.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:10.283+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:11.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:10.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:10.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:10.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:10.763+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:11.565 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:11.279+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:11.565 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:11.279+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:11.565 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:11.279+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:11.565 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:11.279+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:11.566 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:11.211+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:11.566 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:11.211+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:11.566 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:11.211+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:11.566 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:11.212+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:11.837 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:57:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:57:11.565+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 41236 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:57:11.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:11.774+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:11.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:11.774+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:11.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:11.774+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:11.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:11.775+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:12.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:12.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:12.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:12.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:12.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:12.326+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:12.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:12.326+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:12.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:12.236+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:12.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:12.236+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:12.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:12.236+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:12.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:12.236+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:13.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:12.733+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:13.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:12.733+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:13.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:12.733+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:13.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:12.734+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:13.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:13.348+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:13.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:13.348+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:13.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:13.348+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:13.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:13.348+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:13.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:13.219+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:13.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:13.219+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:13.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:13.219+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:13.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:13.220+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:14.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:13.741+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:13.741+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:13.741+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:13.741+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:14.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:14.191+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:14.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:14.191+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:14.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:14.191+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:14.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:14.192+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:14.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:14.386+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:14.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:14.386+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:14.386+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:14.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:14.386+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:15.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:14.766+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:15.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:14.766+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:15.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:14.766+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:15.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:14.767+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:15.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:15.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:15.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:15.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:15.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:15.400+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:15.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:15.401+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:15.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:15.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:15.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:15.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:15.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:15.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:15.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:15.167+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:16.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:15.775+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:16.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:15.775+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:16.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:15.775+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:16.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:15.776+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:16.425 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:16.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:16.425 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:16.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:16.425 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:16.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:16.425 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:16.163+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:16.692 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:57:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:57:16.571+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 41327 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:57:16.693 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:16.426+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:16.693 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:16.426+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:16.693 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:16.426+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:16.693 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:16.427+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:17.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:16.820+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:17.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:16.820+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:17.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:16.820+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:17.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:16.820+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:17.466 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:17.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:17.466 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:17.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:17.467 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:17.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:17.467 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:17.142+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:17.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:17.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:17.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:17.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:17.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:17.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:17.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:17.468+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:18.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:17.801+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:18.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:17.801+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:18.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:17.801+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:18.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:17.802+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:18.494 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:18.113+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:18.494 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:18.113+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:18.494 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:18.113+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:18.495 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:18.113+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:18.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:18.495+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:18.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:18.495+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:18.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:18.495+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:18.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:18.496+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:19.161 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:18.840+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:19.161 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:18.840+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:19.161 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:18.840+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:19.162 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:18.841+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:19.511 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:19.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:19.511 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:19.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:19.511 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:19.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:19.511 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:19.163+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:19.790 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:19.512+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:19.791 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:19.512+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:19.791 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:19.512+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:19.791 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:19.513+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:20.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:19.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:20.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:19.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:20.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:19.791+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:20.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:19.793+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:20.549 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:20.168+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:20.549 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:20.168+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:20.549 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:20.168+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:20.550 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:20.168+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:20.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:20.549+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:20.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:20.549+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:20.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:20.549+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:20.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:20.550+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:20.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:20.779+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:20.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:20.779+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:20.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:20.779+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:20.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:20.780+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:21.520 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:21.181+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:21.520 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:21.181+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:21.520 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:21.181+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:21.520 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:21.181+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:21.776 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:57:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:57:21.575+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 41418 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:57:21.776 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:21.520+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:21.776 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:21.520+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:21.777 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:21.520+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:21.777 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:21.521+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:22.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:21.776+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:21.776+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:21.776+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:21.777+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:22.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:22.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:22.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:22.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:22.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:22.162+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:22.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:22.163+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:22.753 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:22.474+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:22.753 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:22.474+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:22.754 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:22.474+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:22.754 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:22.475+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:23.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:22.753+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:23.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:22.753+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:22.753+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:22.754+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:23.430 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:23.125+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:23.430 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:23.125+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:23.431 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:23.125+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:23.431 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:23.126+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:23.729 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:23.430+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:23.729 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:23.430+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:23.729 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:23.430+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:23.729 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:23.431+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:24.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:23.729+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:24.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:23.729+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:24.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:23.729+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:24.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:23.729+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:24.405 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:24.152+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:24.405 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:24.152+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:24.405 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:24.152+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:24.405 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:24.153+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:24.756 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:24.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:24.756 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:24.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:24.756 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:24.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:24.756 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:24.406+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:25.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:24.756+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:25.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:24.756+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:25.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:24.756+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:25.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:24.757+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:25.396 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:25.128+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:25.396 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:25.128+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:25.396 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:25.128+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:25.396 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:25.129+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:25.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:25.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:25.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:25.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:25.801 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:25.396+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:25.801 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:25.396+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:26.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:25.800+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:26.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:25.800+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:26.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:25.800+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:26.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:25.801+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:26.439 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:26.130+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:26.439 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:26.130+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:26.440 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:26.130+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:54.432616+0000 (oldest deadline 2024-03-19T03:47:14.432616+0000) 2024-03-19T03:57:26.440 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:26.130+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:26.703 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:57:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:57:26.580+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 41509 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:57:26.704 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:26.439+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:26.704 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:26.439+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:26.704 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:26.439+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:26.704 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:26.440+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:27.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:26.781+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:27.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:26.781+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:27.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:26.781+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:27.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:26.782+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:27.405 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:27.103+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:27.799 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:27.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:27.799 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:27.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:27.799 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:27.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:27.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:27.406+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:28.066 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:27.799+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:28.066 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:27.799+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:28.066 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:27.799+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:46:51.922908+0000 (oldest deadline 2024-03-19T03:47:11.922908+0000) 2024-03-19T03:57:28.067 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:27.800+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:28.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:28.066+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:28.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:28.409+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:28.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:28.409+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:28.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:28.409+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:28.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:28.409+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:29.244 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:28.848+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:29.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:29.019+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:29.392+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:29.392+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:29.392+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:29.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:29.393+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:30.249 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:29.862+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:30.249 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:30.019+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:30.531 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:30.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:30.531 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:30.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:30.531 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:30.361+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:30.531 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:30.362+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:31.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:30.911+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:31.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:31.042+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:31.708 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:57:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:57:31.585+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 41601 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:57:31.709 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:31.374+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:31.709 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:31.374+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:31.709 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:31.374+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:31.709 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:31.375+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:32.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:31.875+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:32.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:32.062+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:32.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:32.410+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:32.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:32.410+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:32.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:32.410+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:32.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:32.411+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:33.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:32.862+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:33.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:33.059+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:33.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:33.455+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:33.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:33.455+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:33.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:33.455+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:33.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:33.457+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:34.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:33.822+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:34.045+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:34.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:34.468+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:34.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:34.468+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:34.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:34.468+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:34.801 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:34.469+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:35.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:34.800+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:35.043+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:35.812 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:35.468+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:35.812 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:35.468+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:35.812 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:35.468+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:35.812 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:35.469+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:36.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:35.812+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:36.020+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:36.785 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:57:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:57:36.590+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 41692 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:57:36.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:36.509+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:36.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:36.509+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:36.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:36.509+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:47:04.464907+0000 (oldest deadline 2024-03-19T03:47:24.464907+0000) 2024-03-19T03:57:36.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:36.510+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:37.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:36.785+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:37.002+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:37.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:37.539+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:37.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:37.747+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:38.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:37.973+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:38.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:38.520+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:38.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:38.729+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:39.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:38.936+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:39.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:39.771+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:39.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:39.540+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:40.249 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:39.901+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:40.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:40.794+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:40.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:40.550+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:41.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:40.941+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:41.788 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:57:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:57:41.596+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 41784 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:57:41.789 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:41.511+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:42.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:41.796+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:42.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:41.955+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:42.819 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:42.511+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:43.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:42.955+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:43.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:42.823+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:43.798 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:43.517+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:44.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:43.989+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:44.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:43.798+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:44.755 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:44.505+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:45.030 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:44.756+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:45.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:45.030+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:45.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:45.521+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:45.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:45.762+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:46.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:46.071+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:46.787 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:57:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:57:46.601+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 41875 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:57:46.788 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:46.498+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:47.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:46.787+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:47.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:47.029+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:47.818 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:47.511+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:48.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:47.818+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:48.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:47.998+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:48.812 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:48.558+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:49.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:48.968+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:48.812+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:48.812+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:48.812+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:48.813+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:49.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:49.532+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:50.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:49.797+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:49.797+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:49.797+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:49.798+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:49.975+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:49.975+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:49.975+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:49.975+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:50.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:50.555+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:51.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:51.016+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:51.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:51.016+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:51.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:51.016+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:51.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:51.017+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:51.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:50.825+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:51.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:50.825+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:51.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:50.825+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:51.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:50.826+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:51.874 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:57:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:57:51.606+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 41967 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:57:51.874 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:51.596+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:52.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:52.054+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:52.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:52.054+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:52.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:52.054+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:52.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:52.055+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:52.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:51.874+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:52.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:51.874+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:52.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:51.874+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:52.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:51.875+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:53.026 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:52.615+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:53.027 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:52.846+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:53.027 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:52.846+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:53.027 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:52.846+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:53.027 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:52.847+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:53.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:53.026+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:53.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:53.026+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:53.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:53.026+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:53.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:53.027+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:54.048 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:53.658+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:54.049 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:53.884+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:54.049 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:53.884+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:54.049 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:53.884+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:54.049 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:53.885+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:54.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:54.048+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:54.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:54.048+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:54.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:54.048+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:54.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:54.049+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:55.035 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:54.678+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:55.036 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:54.919+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:55.036 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:54.919+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:55.036 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:54.919+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:55.036 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:54.920+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:55.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:55.035+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:55.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:55.035+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:55.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:55.035+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:55.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:55.036+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:56.073 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:55.681+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:56.073 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:55.895+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:56.073 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:55.895+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:56.073 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:55.895+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:56.073 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:55.897+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:56.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:56.073+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:56.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:56.073+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:56.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:56.073+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:56.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:56.073+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:56.908 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:57:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:57:56.612+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 42058 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:57:56.908 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:56.676+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:57.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:56.908+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:57.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:56.908+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:57.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:56.908+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:57.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:56.909+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:57.060+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:57.060+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:57.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:57.060+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:57.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:57.060+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:58.048 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:57.700+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:58.049 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:57.908+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:58.049 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:57.908+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:58.049 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:57.908+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:58.049 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:57.909+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:58.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:58.048+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:58.048+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:58.048+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:58.049+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:59.020 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:58.735+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:57:59.020 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:58.735+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:57:59.020 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:58.735+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:57:59.020 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:58.736+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:57:59.021 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:58.876+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:59.021 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:58.876+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:59.021 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:58.876+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:57:59.021 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:58.876+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:57:59.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:59.020+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:59.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:59.020+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:59.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:59.020+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:57:59.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:57:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:57:59.020+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:00.049 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:59.689+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:00.049 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:59.689+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:00.050 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:59.689+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:00.050 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:57:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:57:59.690+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:00.050 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:59.902+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:00.050 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:59.902+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:00.050 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:59.902+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:00.050 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:57:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:57:59.903+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:00.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:00.049+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:00.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:00.049+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:00.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:00.049+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:00.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:00.050+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:01.050 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:00.681+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:01.051 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:00.681+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:01.051 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:00.681+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:01.051 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:00.682+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:01.051 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:00.905+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:01.051 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:00.905+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:01.051 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:00.905+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:01.052 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:00.906+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:01.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:01.050+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:01.050+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:01.050+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:01.051+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:01.908 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:58:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:58:01.617+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 42149 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:58:01.908 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:01.667+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:01.908 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:01.667+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:01.909 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:01.667+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:01.909 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:01.668+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:02.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:01.908+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:02.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:01.908+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:02.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:01.908+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:02.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:01.909+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:02.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:02.024+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:02.024+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:02.024+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:02.025+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:03.049 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:02.710+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:03.050 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:02.710+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:03.050 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:02.710+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:03.050 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:02.710+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:03.050 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:02.915+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:03.050 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:02.915+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:03.051 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:02.915+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:03.051 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:02.916+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:03.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:03.049+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:03.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:03.049+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:03.049+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:03.050+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:04.048 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:03.760+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:04.049 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:03.760+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:04.049 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:03.760+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:04.049 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:03.761+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:04.049 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:03.912+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:04.049 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:03.912+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:04.049 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:03.912+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:04.050 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:03.913+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:04.323 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:04.048+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:04.323 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:04.048+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:04.323 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:04.048+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:04.323 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:04.049+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:05.057 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:04.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:05.057 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:04.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:05.058 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:04.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:05.058 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:04.867+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:05.058 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:04.757+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:05.058 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:04.757+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:05.058 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:04.757+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:05.058 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:04.758+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:05.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:05.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:05.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:05.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:05.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:05.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:05.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:05.058+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:06.021 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:05.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:06.021 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:05.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:06.021 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:05.858+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:06.021 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:05.858+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:06.022 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:05.729+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:06.022 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:05.729+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:06.022 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:05.729+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:06.022 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:05.730+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:06.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:06.021+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:06.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:06.021+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:06.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:06.021+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:06.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:06.022+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:06.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:58:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:58:06.622+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 42240 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:58:06.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:06.777+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:06.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:06.777+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:06.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:06.777+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:06.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:06.778+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:06.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:06.820+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:06.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:06.820+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:06.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:06.820+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:06.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:06.821+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:07.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:06.974+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:07.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:06.974+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:07.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:06.974+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:07.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:06.975+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:08.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:07.732+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:08.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:07.732+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:08.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:07.732+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:08.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:07.732+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:08.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:07.790+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:08.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:07.790+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:08.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:07.790+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:07.791+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:07.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:07.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:07.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:07.958+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:09.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:08.751+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:09.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:08.751+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:09.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:08.751+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:09.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:08.751+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:09.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:08.818+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:09.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:08.818+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:09.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:08.818+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:08.819+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:08.923+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:08.923+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:08.923+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:08.923+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:10.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:09.745+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:10.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:09.745+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:10.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:09.745+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:10.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:09.745+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:10.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:09.846+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:10.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:09.846+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:10.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:09.846+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:10.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:09.847+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:09.941+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:09.941+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:09.941+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:09.941+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:11.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:10.706+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:11.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:10.706+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:11.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:10.706+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:11.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:10.707+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:11.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:10.911+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:11.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:10.911+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:10.911+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:10.912+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:11.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:10.804+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:11.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:10.804+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:11.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:10.804+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:11.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:10.805+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:11.893 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:58:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:58:11.628+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 42332 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:58:11.894 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:11.676+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:11.894 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:11.676+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:11.894 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:11.676+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:11.894 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:11.677+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:11.895 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:11.822+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:11.895 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:11.822+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:11.895 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:11.822+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:11.895 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:11.823+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:12.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:11.893+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:12.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:11.893+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:12.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:11.893+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:12.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:11.894+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:12.897+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:12.897+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:12.897+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:12.898+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:12.685+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:12.685+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:12.685+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:12.686+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:12.773+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:13.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:12.773+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:13.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:12.773+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:13.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:12.774+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:13.794 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:13.701+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:13.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:13.701+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:13.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:13.701+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:13.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:13.701+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:13.795+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:13.795+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:13.795+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:13.795+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:14.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:13.901+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:13.901+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:13.901+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:13.902+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:15.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:14.889+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:15.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:14.889+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:15.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:14.889+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:15.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:14.890+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:15.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:14.696+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:15.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:14.696+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:15.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:14.696+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:15.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:14.696+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:15.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:14.843+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:15.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:14.843+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:15.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:14.843+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:15.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:14.843+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:16.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:15.887+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:16.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:15.887+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:16.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:15.887+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:16.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:15.888+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:15.716+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:15.716+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:15.716+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:16.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:15.716+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:15.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:15.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:16.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:15.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:16.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:15.837+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:16.906 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:58:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:58:16.633+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 42423 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:58:16.907 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:16.730+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:16.907 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:16.730+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:16.907 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:16.730+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:16.907 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:16.731+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:16.908 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:16.859+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:16.908 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:16.859+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:16.908 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:16.859+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:16.908 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:16.859+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:17.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:16.907+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:17.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:16.907+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:16.907+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:16.907+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:18.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:17.685+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:18.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:17.685+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:18.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:17.685+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:18.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:17.685+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:18.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:17.921+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:18.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:17.921+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:18.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:17.921+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:18.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:17.921+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:17.898+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:17.898+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:17.898+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:18.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:17.899+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:19.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:18.677+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:19.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:18.677+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:19.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:18.677+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:19.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:18.678+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:19.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:18.857+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:19.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:18.857+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:18.857+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:19.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:18.857+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:19.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:18.908+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:19.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:18.908+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:19.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:18.908+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:19.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:18.909+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:19.901 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:19.814+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:19.901 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:19.814+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:19.901 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:19.814+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:19.901 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:19.815+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:19.902 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:19.645+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:19.902 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:19.645+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:19.902 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:19.645+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:19.902 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:19.646+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:20.251 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:19.901+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:20.251 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:19.901+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:20.251 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:19.901+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:20.252 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:19.902+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:20.878 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:20.612+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:20.879 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:20.612+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:20.879 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:20.612+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:20.879 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:20.612+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:20.879 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:20.805+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:20.879 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:20.805+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:20.879 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:20.805+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:20.879 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:20.805+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:21.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:20.878+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:21.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:20.878+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:21.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:20.878+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:21.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:20.879+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:21.910 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:58:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:58:21.638+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 42514 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:58:21.910 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:21.662+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:21.910 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:21.662+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:21.911 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:21.662+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:21.911 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:21.662+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:21.911 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:21.793+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:21.911 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:21.793+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:21.911 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:21.793+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:21.911 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:21.794+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:22.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:21.910+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:22.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:21.910+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:22.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:21.910+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:22.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:21.910+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:22.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:22.681+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:22.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:22.681+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:22.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:22.681+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:22.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:22.682+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:22.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:22.775+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:22.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:22.775+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:22.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:22.775+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:22.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:22.776+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:23.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:22.949+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:23.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:22.949+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:23.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:22.949+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:23.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:22.950+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:24.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:23.730+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:24.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:23.730+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:24.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:23.730+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:24.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:23.731+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:24.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:23.729+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:24.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:23.729+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:24.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:23.729+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:24.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:23.730+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:23.928+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:23.928+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:23.928+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:23.929+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:25.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:24.716+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:25.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:24.716+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:25.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:24.716+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:25.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:24.717+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:25.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:24.755+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:25.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:24.755+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:25.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:24.755+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:25.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:24.755+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:24.922+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:24.922+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:24.922+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:24.923+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:26.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:25.780+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:26.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:25.780+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:26.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:25.780+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:26.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:25.780+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:26.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:25.681+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:26.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:25.681+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:26.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:25.681+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:26.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:25.682+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:25.895+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:25.895+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:25.895+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:25.896+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:27.087 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:58:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:58:26.643+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 42605 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:58:27.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:26.769+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:27.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:26.769+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:27.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:26.769+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:27.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:26.770+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:27.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:26.889+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:27.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:26.889+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:27.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:26.890+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:27.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:26.890+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:27.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:26.721+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:27.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:26.721+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:27.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:26.721+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:27.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:26.722+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:28.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:27.810+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:28.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:27.810+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:28.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:27.810+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:28.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:27.811+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:27.734+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:27.734+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:28.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:27.734+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:28.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:27.735+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:27.841+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:27.841+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:27.841+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:27.841+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:29.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:28.786+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:29.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:28.786+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:29.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:28.786+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:29.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:28.787+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:29.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:28.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:29.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:28.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:29.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:28.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:28.770+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:29.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:28.853+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:29.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:28.853+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:29.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:28.853+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:29.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:28.854+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:30.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:29.807+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:30.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:29.807+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:30.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:29.807+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:30.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:29.808+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:30.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:29.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:30.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:29.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:30.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:29.769+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:30.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:29.769+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:30.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:29.812+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:30.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:29.812+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:30.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:29.812+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:30.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:29.813+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:31.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:30.854+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:31.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:30.854+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:31.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:30.854+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:31.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:30.854+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:31.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:30.854+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:30.854+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:30.854+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:31.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:30.855+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:30.818+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:30.818+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:31.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:30.818+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:31.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:30.819+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:31.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:58:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:58:31.649+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 42697 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:58:31.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:31.843+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:31.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:31.843+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:31.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:31.843+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:31.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:31.844+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:31.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:31.861+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:31.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:31.861+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:31.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:31.861+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:31.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:31.861+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:31.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:31.831+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:31.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:31.831+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:31.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:31.831+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:31.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:31.831+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:33.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:32.830+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:33.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:32.830+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:33.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:32.830+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:33.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:32.830+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:32.863+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:32.863+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:32.863+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:33.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:32.864+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:32.882+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:32.882+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:32.882+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:33.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:32.882+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:34.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:33.818+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:34.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:33.818+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:34.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:33.818+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:34.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:33.819+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:33.876+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:33.876+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:34.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:33.876+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:34.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:33.877+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:34.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:33.871+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:34.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:33.871+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:34.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:33.871+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:34.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:33.872+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:35.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:34.773+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:35.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:34.773+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:35.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:34.773+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:35.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:34.773+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:34.884+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:35.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:34.884+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:35.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:34.884+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:35.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:34.885+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:35.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:34.919+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:35.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:34.919+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:35.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:34.919+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:35.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:34.919+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:36.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:35.727+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:36.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:35.727+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:36.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:35.727+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:36.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:35.727+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:36.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:35.910+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:36.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:35.910+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:36.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:35.910+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:35.910+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:35.880+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:35.880+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:35.880+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:35.881+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:37.087 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:58:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:58:36.656+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 42788 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:58:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:36.862+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:36.862+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:36.862+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:36.863+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:37.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:36.682+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:37.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:36.682+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:37.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:36.682+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:37.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:36.682+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:37.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:36.871+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:37.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:36.871+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:37.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:36.871+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:37.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:36.872+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:38.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:37.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:38.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:37.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:38.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:37.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:38.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:37.866+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:38.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:37.691+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:38.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:37.691+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:38.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:37.691+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:38.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:37.693+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:38.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:37.883+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:38.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:37.883+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:38.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:37.883+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:38.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:37.884+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:39.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:38.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:39.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:38.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:39.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:38.827+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:39.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:38.828+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:39.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:38.698+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:39.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:38.698+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:39.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:38.698+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:39.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:38.699+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:38.892+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:38.892+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:38.892+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:38.893+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:39.936 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:39.667+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:39.937 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:39.667+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:39.937 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:39.667+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:39.937 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:39.668+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:39.937 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:39.876+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:39.937 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:39.876+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:39.938 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:39.876+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:39.938 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:39.877+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:40.252 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:39.937+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:40.253 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:39.937+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:40.253 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:39.937+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:40.253 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:39.938+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:40.942 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:40.687+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:40.943 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:40.687+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:40.943 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:40.687+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:40.943 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:40.688+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:40.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:40.840+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:40.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:40.840+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:40.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:40.840+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:40.944 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:40.840+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:41.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:40.942+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:41.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:40.942+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:41.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:40.942+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:41.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:40.943+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:41.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:58:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:58:41.660+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 42879 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:58:41.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:41.804+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:41.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:41.804+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:41.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:41.804+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:41.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:41.805+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:41.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:41.698+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:41.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:41.698+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:41.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:41.698+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:41.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:41.699+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:42.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:41.973+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:42.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:41.973+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:42.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:41.973+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:42.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:41.974+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:42.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:42.701+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:42.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:42.701+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:42.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:42.701+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:42.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:42.702+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:42.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:42.845+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:42.953 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:42.845+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:42.953 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:42.845+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:42.953 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:42.845+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:43.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:42.952+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:43.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:42.952+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:42.952+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:42.953+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:43.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:43.655+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:43.931 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:43.655+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:43.931 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:43.656+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:43.931 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:43.656+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:43.931 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:43.846+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:43.931 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:43.846+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:43.931 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:43.846+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:43.932 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:43.846+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:44.259 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:43.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:44.259 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:43.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:44.259 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:43.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:44.259 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:43.931+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:45.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:44.820+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:45.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:44.820+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:45.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:44.820+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:45.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:44.821+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:44.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:44.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:45.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:44.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:45.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:44.661+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:44.898+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:44.898+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:44.898+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:44.899+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:45.933 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:45.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:45.933 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:45.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:45.933 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:45.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:45.933 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:45.642+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:45.934 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:45.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:45.934 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:45.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:45.934 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:45.794+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:45.934 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:45.794+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:46.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:45.933+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:46.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:45.933+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:46.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:45.933+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:46.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:45.933+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:46.956 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:58:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:58:46.666+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 42970 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:58:46.956 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:46.664+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:46.957 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:46.664+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:46.957 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:46.664+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:46.957 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:46.665+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:46.957 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:46.778+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:46.957 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:46.778+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:46.957 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:46.778+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:46.958 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:46.780+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:47.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:46.956+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:47.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:46.956+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:47.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:46.956+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:47.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:46.957+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:47.928 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:47.617+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:47.928 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:47.617+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:47.928 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:47.617+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:47.928 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:47.618+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:47.929 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:47.777+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:47.929 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:47.777+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:47.929 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:47.777+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:47.929 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:47.778+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:48.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:47.928+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:48.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:47.928+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:48.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:47.928+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:48.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:47.928+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:48.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:48.808+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:48.953 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:48.808+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:48.953 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:48.808+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:48.953 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:48.809+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:48.953 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:48.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:48.953 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:48.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:48.953 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:48.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:48.954 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:48.596+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:49.244 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:48.952+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:49.244 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:48.952+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:49.244 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:48.952+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:49.244 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:48.953+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:49.927 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:49.616+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:49.928 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:49.616+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:49.928 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:49.616+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:49.928 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:49.617+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:49.928 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:49.854+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:49.928 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:49.854+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:49.929 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:49.854+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:49.929 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:49.855+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:50.253 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:49.927+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:50.254 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:49.927+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:50.254 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:49.927+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:50.254 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:49.928+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:50.899 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:50.601+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:50.900 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:50.601+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:50.900 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:50.601+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:50.900 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:50.601+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:51.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:50.965+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:51.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:50.965+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:51.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:50.965+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:51.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:50.966+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:51.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:50.899+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:51.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:50.899+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:51.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:50.899+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:51.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:50.900+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:51.837 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:58:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:58:51.671+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 43062 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:58:51.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:51.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:51.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:51.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:51.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:51.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:51.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:51.568+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:51.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:51.923+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:51.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:51.923+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:51.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:51.923+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:51.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:51.925+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:52.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:51.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:52.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:51.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:52.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:51.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:52.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:51.958+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:52.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:52.583+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:52.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:52.583+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:52.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:52.583+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:52.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:52.583+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:53.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:52.939+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:53.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:52.939+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:53.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:52.939+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:53.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:52.939+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:53.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:52.937+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:53.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:52.937+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:53.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:52.937+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:53.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:52.938+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:53.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:53.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:53.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:53.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:53.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:53.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:53.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:53.568+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:54.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:53.961+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:54.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:53.961+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:54.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:53.961+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:54.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:53.961+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:54.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:53.945+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:54.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:53.945+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:54.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:53.945+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:54.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:53.945+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:54.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:54.613+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:54.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:54.613+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:54.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:54.613+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:54.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:54.614+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:54.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:54.950+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:54.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:54.950+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:55.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:54.950+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:55.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:54.950+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:55.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:54.975+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:55.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:54.975+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:55.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:54.975+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:55.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:54.976+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:55.961 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:55.644+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:55.962 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:55.644+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:55.962 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:55.644+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:55.962 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:55.645+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:56.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:55.961+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:56.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:55.961+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:56.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:55.961+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:56.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:55.962+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:56.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:55.998+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:56.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:55.998+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:56.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:55.998+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:56.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:55.999+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:56.916 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:58:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:58:56.677+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 43153 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:58:56.916 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:56.625+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:56.916 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:56.625+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:56.916 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:56.625+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:56.917 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:56.626+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:57.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:56.916+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:57.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:56.916+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:57.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:56.916+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:57.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:56.917+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:57.031+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:57.031+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:57.031+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:57.032+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:57.907 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:57.624+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:57.907 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:57.624+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:57.907 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:57.624+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:57.908 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:57.625+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:58.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:58.062+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:58.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:58.062+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:58.062+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:58.062+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:58.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:57.907+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:58.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:57.907+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:58.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:57.907+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:58.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:57.908+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:58.910 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:58.620+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:58.911 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:58.620+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:58.911 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:58.620+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:58.911 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:58.620+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:59.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:58.911+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:59.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:58.911+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:59.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:58.911+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:58:59.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:58.912+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:58:59.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:59.079+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:59.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:59.079+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:59.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:59.079+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:58:59.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:58:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:58:59.080+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:58:59.903 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:59.629+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:59.903 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:59.629+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:59.904 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:59.629+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:58:59.904 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:58:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:58:59.631+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:00.254 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:59.903+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:00.255 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:59.903+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:00.255 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:59.903+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:00.255 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:58:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:58:59.904+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:00.255 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:00.048+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:00.255 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:00.048+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:00.256 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:00.048+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:00.256 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:00.049+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:00.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:00.583+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:00.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:00.583+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:00.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:00.583+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:00.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:00.584+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:01.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:01.059+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:01.059+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:01.059+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:01.060+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:01.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:00.919+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:01.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:00.919+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:01.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:00.919+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:01.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:00.920+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:01.793 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:59:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:59:01.683+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 43244 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:59:01.794 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:01.537+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:01.794 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:01.537+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:01.794 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:01.537+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:01.794 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:01.538+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:02.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:01.934+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:02.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:01.934+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:02.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:01.934+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:02.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:01.935+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:02.037+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:02.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:02.037+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:02.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:02.037+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:02.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:02.037+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:02.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:02.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:02.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:02.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:02.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:02.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:02.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:02.489+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:03.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:02.931+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:03.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:02.931+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:03.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:02.931+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:03.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:02.932+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:03.086+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:03.086+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:03.086+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:03.087+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:03.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:03.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:03.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:03.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:03.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:03.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:03.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:03.468+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:04.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:03.900+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:04.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:03.900+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:04.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:03.900+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:04.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:03.901+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:04.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:04.066+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:04.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:04.066+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:04.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:04.066+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:04.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:04.066+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:04.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:04.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:04.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:04.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:04.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:04.454+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:04.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:04.455+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:05.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:04.859+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:05.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:04.859+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:05.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:04.859+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:05.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:04.859+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:05.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:05.036+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:05.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:05.036+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:05.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:05.036+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:05.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:05.037+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:05.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:05.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:05.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:05.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:05.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:05.467+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:05.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:05.469+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:06.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:06.058+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:06.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:06.058+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:06.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:06.058+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:06.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:06.059+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:06.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:05.902+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:06.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:05.902+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:06.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:05.902+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:06.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:05.903+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:06.810 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:59:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:59:06.688+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 43335 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:59:06.811 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:06.484+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:06.811 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:06.484+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:06.811 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:06.484+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:06.811 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:06.485+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:07.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:06.896+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:07.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:06.896+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:07.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:06.896+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:07.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:06.897+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:07.510 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:07.094+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:07.510 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:07.094+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:07.510 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:07.094+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:07.510 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:07.095+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:07.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:07.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:07.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:07.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:07.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:07.510+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:07.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:07.511+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:08.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:08.049+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:08.049+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:08.049+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:08.050+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:08.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:07.852+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:08.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:07.852+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:08.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:07.852+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:08.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:07.853+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:08.826 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:08.548+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:08.826 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:08.548+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:08.827 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:08.548+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:08.827 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:08.549+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:08.826+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:09.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:08.826+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:09.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:08.826+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:09.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:08.827+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:09.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:09.069+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:09.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:09.069+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:09.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:09.069+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:09.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:09.069+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:10.056 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:09.591+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:10.057 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:09.591+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:10.057 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:09.591+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:10.057 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:09.592+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:10.057 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:09.813+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:10.057 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:09.813+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:10.058 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:09.813+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:10.058 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:09.814+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:10.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:10.056+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:10.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:10.056+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:10.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:10.056+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:10.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:10.057+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:11.015 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:10.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:11.015 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:10.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:11.015 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:10.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:11.015 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:10.642+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:11.015 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:10.859+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:11.016 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:10.859+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:11.016 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:10.859+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:11.016 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:10.859+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:11.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:11.015+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:11.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:11.015+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:11.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:11.015+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:11.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:11.016+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:11.907 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:59:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:59:11.694+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 43427 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:59:11.907 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:11.834+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:11.907 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:11.834+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:11.907 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:11.834+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:11.908 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:11.835+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:11.908 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:11.632+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:11.908 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:11.632+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:11.908 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:11.632+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:11.908 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:11.633+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:12.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:11.986+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:12.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:11.986+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:12.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:11.986+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:12.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:11.987+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:12.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:12.792+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:12.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:12.792+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:12.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:12.792+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:12.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:12.793+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:12.673+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:12.673+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:12.673+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:12.673+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:13.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:12.949+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:13.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:12.949+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:13.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:12.949+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:13.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:12.950+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:13.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:13.800+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:13.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:13.800+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:13.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:13.800+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:13.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:13.801+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:13.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:13.663+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:13.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:13.663+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:13.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:13.663+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:13.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:13.663+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:14.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:13.973+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:14.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:13.973+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:14.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:13.973+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:14.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:13.974+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:14.988 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:14.629+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:14.988 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:14.629+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:14.988 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:14.629+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:14.988 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:14.630+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:14.989 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:14.838+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:14.989 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:14.838+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:14.989 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:14.838+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:14.989 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:14.839+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:15.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:14.988+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:15.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:14.988+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:15.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:14.988+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:15.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:14.988+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:16.013 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:15.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:16.013 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:15.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:16.014 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:15.836+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:16.014 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:15.837+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:16.014 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:15.601+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:16.014 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:15.601+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:16.014 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:15.601+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:16.014 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:15.602+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:16.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:16.013+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:16.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:16.013+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:16.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:16.013+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:16.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:16.014+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:17.013 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:59:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:59:16.699+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 43518 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:59:17.014 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:16.803+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:17.014 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:16.803+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:17.015 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:16.803+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:17.015 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:16.804+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:17.015 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:16.596+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:17.016 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:16.596+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:17.016 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:16.596+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:17.016 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:16.596+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:17.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:17.013+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:17.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:17.013+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:17.013+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:17.014+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:18.042 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:17.639+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:18.042 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:17.639+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:18.042 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:17.639+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:18.042 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:17.640+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:18.043 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:17.760+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:18.043 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:17.760+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:18.043 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:17.760+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:18.043 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:17.761+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:18.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:18.042+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:18.457 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:18.042+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:18.458 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:18.042+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:18.458 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:18.043+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:19.008 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:18.607+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:19.008 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:18.607+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:19.008 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:18.607+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:19.008 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:18.608+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:19.008 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:18.764+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:19.009 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:18.764+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:19.009 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:18.764+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:19.009 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:18.765+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:19.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:19.007+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:19.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:19.007+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:19.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:19.007+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:19.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:19.008+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:20.006 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:19.802+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:20.006 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:19.802+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:20.007 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:19.802+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:20.007 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:19.803+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:20.007 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:19.650+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:20.007 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:19.650+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:20.007 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:19.650+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:20.007 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:19.651+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:20.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:20.006+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:20.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:20.006+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:20.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:20.006+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:20.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:20.007+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:20.967 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:20.848+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:20.968 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:20.848+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:20.968 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:20.848+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:20.968 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:20.849+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:20.968 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:20.616+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:20.968 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:20.616+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:20.969 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:20.616+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:20.969 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:20.617+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:21.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:20.967+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:21.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:20.967+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:21.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:20.967+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:21.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:20.968+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:21.935 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:59:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:59:21.705+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 43610 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:59:21.936 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:21.846+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:21.936 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:21.846+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:21.936 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:21.846+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:21.936 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:21.847+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:21.937 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:21.632+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:21.937 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:21.632+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:21.937 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:21.632+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:21.937 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:21.633+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:22.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:21.936+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:22.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:21.936+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:22.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:21.936+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:22.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:21.936+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:23.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:22.667+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:23.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:22.667+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:23.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:22.667+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:23.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:22.668+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:22.861+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:22.861+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:22.861+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:22.862+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:23.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:22.889+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:23.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:22.889+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:23.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:22.889+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:23.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:22.890+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:23.899 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:23.627+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:23.899 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:23.627+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:23.899 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:23.627+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:23.899 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:23.627+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:23.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:23.851+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:23.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:23.851+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:23.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:23.851+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:23.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:23.852+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:24.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:23.899+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:24.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:23.899+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:24.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:23.899+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:24.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:23.900+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:24.923 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:24.668+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:24.923 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:24.668+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:24.923 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:24.668+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:24.923 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:24.669+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:24.924 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:24.825+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:24.924 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:24.825+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:24.924 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:24.825+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:24.924 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:24.826+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:25.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:24.923+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:25.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:24.923+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:25.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:24.923+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:25.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:24.924+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:26.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:25.854+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:26.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:25.854+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:26.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:25.854+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:26.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:25.855+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:26.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:25.663+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:26.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:25.663+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:26.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:25.663+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:26.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:25.664+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:25.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:25.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:25.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:25.907+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:26.925 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:59:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:59:26.710+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 43701 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:59:26.926 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:26.628+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:26.926 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:26.628+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:26.926 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:26.628+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:26.926 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:26.629+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:26.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:26.869+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:26.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:26.869+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:26.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:26.869+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:26.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:26.870+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:27.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:26.925+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:26.925+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:26.925+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:26.926+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:28.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:27.659+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:28.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:27.659+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:28.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:27.659+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:28.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:27.660+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:27.851+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:27.851+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:27.851+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:28.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:27.851+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:27.878+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:27.878+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:27.878+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:27.879+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:29.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:28.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:28.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:28.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:28.859+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:29.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:28.656+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:29.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:28.656+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:29.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:28.656+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:29.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:28.657+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:28.885+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:28.885+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:28.885+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:29.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:28.886+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:30.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:29.879+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:29.879+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:29.879+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:29.879+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:30.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:29.667+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:30.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:29.667+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:30.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:29.667+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:30.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:29.668+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:30.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:29.862+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:30.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:29.862+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:30.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:29.862+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:30.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:29.864+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:30.929 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:30.674+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:30.929 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:30.674+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:30.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:30.674+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:30.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:30.675+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:30.930 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:30.830+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:30.930 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:30.830+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:30.930 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:30.830+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:30.930 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:30.831+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:30.931 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:30.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:30.931 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:30.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:30.931 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:30.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:31.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:30.930+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:31.941 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:59:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:59:31.716+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 43787 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:59:31.942 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:31.668+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:31.942 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:31.668+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:31.942 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:31.668+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:31.942 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:31.669+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:31.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:31.877+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:31.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:31.877+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:31.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:31.877+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:31.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:31.878+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:32.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:31.942+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:32.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:31.942+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:32.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:31.942+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:32.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:31.943+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:32.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:32.900+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:32.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:32.900+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:32.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:32.900+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:32.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:32.901+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:32.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:32.688+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:32.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:32.688+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:32.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:32.688+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:32.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:32.689+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:33.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:32.972+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:33.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:32.972+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:33.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:32.972+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:33.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:32.973+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:34.010 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:33.664+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:34.010 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:33.664+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:34.010 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:33.664+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:34.010 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:33.666+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:34.011 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:33.851+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:34.011 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:33.851+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:34.011 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:33.851+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:34.011 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:33.852+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:34.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:34.010+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:34.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:34.010+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:34.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:34.010+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:34.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:34.011+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:34.983 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:34.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:34.984 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:34.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:34.984 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:34.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:34.984 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:34.661+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:34.984 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:34.805+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:34.984 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:34.805+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:34.984 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:34.805+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:34.984 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:34.806+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:35.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:34.983+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:35.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:34.983+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:35.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:34.983+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:35.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:34.984+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:36.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:35.704+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:36.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:35.704+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:36.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:35.704+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:36.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:35.704+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:36.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:35.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:36.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:35.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:35.847+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:35.848+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:35.950+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:35.950+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:35.950+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:35.951+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:36.994 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:59:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:59:36.721+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 43872 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:59:36.994 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:36.727+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:36.994 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:36.727+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:36.995 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:36.727+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:36.995 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:36.728+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:36.995 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:36.888+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:36.995 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:36.888+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:36.995 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:36.888+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:36.995 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:36.889+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:37.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:36.994+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:37.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:36.994+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:37.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:36.994+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:37.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:36.995+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:37.981 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:37.885+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:37.981 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:37.885+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:37.981 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:37.885+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:37.981 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:37.885+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:37.982 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:37.692+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:37.982 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:37.692+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:37.982 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:37.692+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:37.982 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:37.692+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:38.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:37.981+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:38.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:37.981+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:37.981+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:37.982+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:38.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:38.649+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:38.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:38.649+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:38.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:38.649+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:38.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:38.650+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:38.945 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:38.848+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:38.945 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:38.848+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:38.945 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:38.848+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:38.945 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:38.848+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:38.945 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:38.943+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:38.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:38.944+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:39.244 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:38.944+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:39.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:38.945+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:39.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:39.659+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:39.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:39.659+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:39.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:39.659+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:39.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:39.660+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:39.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:39.895+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:39.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:39.895+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:39.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:39.895+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:39.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:39.896+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:40.256 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:39.975+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:40.256 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:39.975+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:40.257 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:39.975+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:40.257 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:39.976+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:40.967 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:40.689+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:40.968 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:40.689+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:40.968 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:40.689+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:40.968 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:40.690+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:40.968 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:40.894+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:40.969 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:40.894+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:40.969 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:40.894+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:40.969 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:40.896+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:41.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:40.968+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:41.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:40.968+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:41.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:40.968+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:41.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:40.968+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:41.919 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:59:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:59:41.727+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 43954 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:59:41.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:41.658+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:41.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:41.658+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:41.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:41.658+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:41.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:41.659+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:42.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:41.919+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:41.919+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:41.919+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:41.919+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:42.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:41.939+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:42.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:41.939+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:42.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:41.939+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:42.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:41.940+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:42.905 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:42.670+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:42.905 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:42.670+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:42.906 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:42.671+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:42.906 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:42.671+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:43.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:42.958+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:42.958+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:42.958+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:42.958+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:43.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:42.905+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:43.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:42.905+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:43.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:42.905+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:43.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:42.906+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:43.937 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:43.675+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:43.937 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:43.675+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:43.938 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:43.675+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:43.938 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:43.676+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:44.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:43.994+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:44.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:43.994+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:43.994+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:43.994+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:43.937+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:43.937+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:43.937+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:43.938+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:45.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:44.698+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:44.698+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:44.698+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:44.699+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:45.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:44.903+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:45.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:44.903+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:45.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:44.903+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:45.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:44.904+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:44.947+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:44.947+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:44.947+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:45.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:44.948+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:45.961 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:45.697+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:45.962 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:45.697+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:45.962 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:45.697+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:45.962 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:45.698+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:45.963 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:45.868+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:45.963 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:45.868+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:45.963 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:45.868+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:45.963 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:45.869+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:46.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:45.962+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:46.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:45.962+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:46.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:45.962+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:46.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:45.963+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:47.004 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:59:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:59:46.732+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 44036 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:59:47.004 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:46.712+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:47.004 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:46.712+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:47.004 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:46.712+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:47.004 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:46.713+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:47.005 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:46.845+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:47.005 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:46.845+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:47.005 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:46.845+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:47.005 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:46.846+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:47.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:47.004+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:47.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:47.004+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:47.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:47.004+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:47.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:47.005+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:48.027 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:47.678+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:48.027 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:47.678+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:48.027 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:47.678+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:48.027 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:47.679+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:48.028 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:47.814+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:48.028 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:47.814+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:48.028 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:47.814+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:48.028 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:47.815+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:48.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:48.027+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:48.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:48.027+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:48.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:48.027+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:48.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:48.027+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:49.050 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:48.699+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:49.050 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:48.699+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:49.050 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:48.699+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:49.051 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:48.699+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:49.053 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:48.779+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:49.054 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:48.779+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:49.054 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:48.779+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:49.054 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:48.780+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:49.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:49.050+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:49.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:49.050+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:49.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:49.050+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:49.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:49.051+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:50.032 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:49.689+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:50.032 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:49.689+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:50.032 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:49.689+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:50.032 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:49.689+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:50.032 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:49.736+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:50.033 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:49.736+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:50.033 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:49.736+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:50.033 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:49.737+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:50.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:50.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:50.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:50.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:50.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:50.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:50.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:50.033+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:51.047 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:50.675+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:51.048 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:50.675+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:51.048 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:50.675+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:51.048 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:50.675+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:51.048 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:50.734+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:51.048 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:50.734+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:51.048 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:50.734+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:51.049 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:50.735+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:51.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:51.047+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:51.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:51.047+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:51.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:51.047+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:51.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:51.048+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:52.019 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:59:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:59:51.738+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 44125 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:59:52.020 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:51.674+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:52.020 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:51.674+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:52.020 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:51.674+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:52.020 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:51.674+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:52.020 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:51.697+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:52.021 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:51.697+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:52.021 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:51.697+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:52.021 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:51.697+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:52.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:52.019+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:52.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:52.019+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:52.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:52.019+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:52.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:52.020+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:53.063 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:52.684+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:53.063 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:52.684+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:53.064 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:52.684+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:53.064 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:52.685+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:53.064 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:52.647+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:53.064 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:52.647+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:53.064 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:52.647+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:53.064 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:52.648+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:53.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:53.063+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:53.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:53.063+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:53.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:53.063+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:53.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:53.064+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:54.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:53.644+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:54.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:53.644+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:54.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:53.644+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:54.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:53.644+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:54.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:53.652+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:54.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:53.652+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:54.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:53.652+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:54.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:53.653+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:54.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:54.097+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:54.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:54.097+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:54.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:54.097+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:54.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:54.098+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:55.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:54.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:55.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:54.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:55.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:54.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:55.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:54.596+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:55.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:54.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:55.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:54.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:55.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:54.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:55.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:54.628+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:55.574 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:55.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:55.574 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:55.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:55.575 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:55.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:55.575 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:55.107+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:55.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:55.575+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:55.575+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:55.575+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:55.576+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:55.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:55.602+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:55.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:55.602+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:55.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:55.602+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:55.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:55.603+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:56.533 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:56.097+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:56.533 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:56.097+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:56.533 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:56.097+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:56.533 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:56.097+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:56.837 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 03:59:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T03:59:56.745+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 44213 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T03:59:56.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:56.534+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:56.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:56.534+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:56.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:56.534+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:56.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:56.535+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:56.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:56.571+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:56.571+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:56.571+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:56.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:56.572+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:57.521 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:57.135+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:57.522 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:57.135+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:57.522 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:57.135+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:57.522 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:57.135+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:57.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:57.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:57.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:57.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:57.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:57.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:57.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:57.567+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:57.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:57.522+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:57.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:57.522+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:57.522+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:57.523+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:58.499 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:58.165+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:58.500 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:58.165+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:58.500 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:58.165+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:58.500 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:58.165+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:58.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:58.532+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:58.532+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:58.532+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:58.533+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:58.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:58.500+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:58.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:58.500+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:58.500+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:58.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:58.501+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:59.480 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:59.161+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:59.480 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:59.161+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:59.480 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:59.161+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T03:59:59.480 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 03:59:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T03:59:59.162+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T03:59:59.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:59.481+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:59.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:59.481+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:59.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:59.481+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T03:59:59.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 03:59:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T03:59:59.482+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T03:59:59.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:59.554+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:59.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:59.554+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:59.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:59.554+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T03:59:59.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 03:59:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T03:59:59.554+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:00.490 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:00.144+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:00.491 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:00.144+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:00.491 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:00.144+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:00.491 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:00.144+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:00.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:00.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:00.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:00.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:00.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:00.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:00.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:00.567+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:00.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:00.491+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:00.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:00.491+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:00.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:00.491+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:00.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:00.492+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:01.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:01.143+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:01.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:01.143+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:01.143+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:01.143+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:01.749 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:01.559+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:01.750 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:01.559+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:01.750 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:01.559+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:01.750 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:01.560+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:01.750 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:01.479+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:01.750 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:01.479+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:01.750 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:01.479+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:01.750 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:01.480+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:02.087 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:00:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:00:01.749+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 44304 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:00:02.508 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:02.114+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:02.508 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:02.114+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:02.508 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:02.114+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:02.509 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:02.115+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:02.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:02.523+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:02.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:02.523+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:02.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:02.523+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:02.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:02.525+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:02.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:02.508+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:02.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:02.508+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:02.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:02.508+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:02.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:02.509+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:03.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:03.077+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:03.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:03.077+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:03.077+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:03.078+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:03.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:03.512+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:03.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:03.512+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:03.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:03.512+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:03.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:03.513+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:03.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:03.515+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:03.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:03.515+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:03.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:03.515+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:03.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:03.516+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:04.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:04.031+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:04.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:04.031+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:04.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:04.031+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:04.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:04.032+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:04.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:04.525+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:04.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:04.525+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:04.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:04.525+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:04.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:04.526+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:04.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:04.551+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:04.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:04.551+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:04.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:04.551+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:04.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:04.552+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:05.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:05.012+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:05.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:05.012+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:05.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:05.012+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:05.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:05.013+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:05.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:05.533+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:05.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:05.533+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:05.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:05.533+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:05.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:05.534+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:05.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:05.523+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:05.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:05.523+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:05.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:05.523+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:05.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:05.524+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:06.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:06.037+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:06.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:06.037+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:06.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:06.037+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:06.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:06.038+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:06.774 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:00:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:00:06.755+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 44396 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:00:06.774 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:06.557+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:06.775 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:06.557+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:06.775 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:06.557+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:06.775 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:06.557+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:06.775 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:06.510+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:06.775 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:06.510+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:06.775 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:06.510+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:06.776 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:06.511+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:07.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:07.030+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:07.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:07.030+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:07.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:07.030+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:07.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:07.030+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:07.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:07.556+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:07.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:07.556+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:07.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:07.556+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:07.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:07.558+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:07.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:07.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:07.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:07.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:07.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:07.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:07.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:07.589+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:08.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:08.047+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:08.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:08.047+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:08.047+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:08.048+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:08.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:08.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:08.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:08.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:08.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:08.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:08.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:08.572+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:08.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:08.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:08.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:08.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:08.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:08.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:08.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:08.604+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:09.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:09.056+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:09.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:09.056+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:09.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:09.056+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:09.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:09.056+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:09.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:09.616+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:09.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:09.616+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:09.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:09.616+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:09.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:09.616+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:09.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:09.572+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:09.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:09.572+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:09.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:09.572+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:09.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:09.573+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:10.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:10.050+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:10.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:10.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:10.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:10.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:10.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:10.051+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:10.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:10.543+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:10.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:10.543+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:10.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:10.543+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:10.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:10.544+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:10.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:10.653+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:10.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:10.653+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:10.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:10.653+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:10.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:10.653+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:11.518 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:11.090+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:11.518 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:11.090+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:11.518 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:11.090+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:11.518 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:11.091+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:11.770 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:00:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:00:11.760+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 44487 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:00:11.771 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:11.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:11.771 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:11.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:11.771 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:11.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:11.772 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:11.609+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:11.772 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:11.518+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:11.773 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:11.518+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:11.773 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:11.518+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:11.773 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:11.519+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:12.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:12.045+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:12.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:12.045+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:12.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:12.045+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:12.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:12.046+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:12.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:12.624+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:12.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:12.624+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:12.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:12.624+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:12.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:12.625+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:12.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:12.493+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:12.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:12.493+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:12.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:12.493+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:12.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:12.494+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:13.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:13.076+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:13.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:13.076+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:13.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:13.076+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:13.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:13.077+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:13.608 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:13.453+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:13.608 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:13.453+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:13.608 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:13.453+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:13.608 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:13.454+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:14.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:13.608+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:14.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:13.608+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:14.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:13.608+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:14.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:13.608+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:14.492 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:14.112+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:14.493 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:14.112+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:14.493 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:14.112+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:14.493 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:14.113+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:14.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:14.655+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:14.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:14.655+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:14.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:14.655+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:14.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:14.656+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:14.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:14.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:14.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:14.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:14.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:14.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:14.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:14.493+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:15.507 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:15.116+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:15.507 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:15.116+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:15.507 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:15.116+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:15.507 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:15.117+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:15.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:15.693+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:15.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:15.693+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:15.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:15.693+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:15.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:15.694+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:15.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:15.506+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:15.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:15.506+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:15.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:15.506+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:15.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:15.507+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:16.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:16.128+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:16.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:16.128+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:16.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:16.128+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:16.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:16.129+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:16.837 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:00:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:00:16.766+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 44577 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:00:16.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:16.551+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:16.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:16.551+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:16.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:16.551+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:16.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:16.552+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:16.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:16.669+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:16.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:16.669+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:16.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:16.669+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:16.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:16.670+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:17.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:17.088+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:17.088+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:17.088+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:17.089+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:17.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:17.692+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:17.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:17.692+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:17.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:17.692+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:17.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:17.693+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:17.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:17.536+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:17.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:17.536+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:17.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:17.536+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:17.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:17.537+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:18.070+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:18.070+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:18.070+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:18.071+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:18.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:18.741+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:18.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:18.741+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:18.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:18.741+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:18.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:18.741+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:18.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:18.551+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:18.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:18.551+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:18.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:18.551+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:18.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:18.552+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:19.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:19.104+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:19.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:19.104+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:19.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:19.104+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:19.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:19.104+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:19.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:19.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:19.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:19.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:19.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:19.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:19.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:19.705+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:19.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:19.546+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:19.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:19.546+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:19.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:19.546+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:19.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:19.547+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:20.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:20.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:20.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:20.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:20.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:20.132+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:20.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:20.133+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:20.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:20.570+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:20.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:20.570+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:20.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:20.570+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:20.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:20.571+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:20.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:20.666+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:20.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:20.666+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:20.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:20.666+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:20.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:20.667+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:21.539 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:21.145+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:21.539 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:21.145+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:21.539 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:21.145+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:21.539 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:21.146+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:21.791 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:00:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:00:21.771+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 44655 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:00:21.791 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:21.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:21.791 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:21.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:21.791 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:21.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:21.792 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:21.706+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:21.792 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:21.539+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:21.792 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:21.539+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:21.792 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:21.539+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:21.792 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:21.540+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:22.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:22.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:22.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:22.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:22.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:22.167+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:22.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:22.168+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:22.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:22.685+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:22.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:22.685+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:22.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:22.685+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:22.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:22.685+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:22.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:22.547+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:22.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:22.547+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:22.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:22.547+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:22.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:22.548+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:23.519 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:23.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:23.520 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:23.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:23.520 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:23.141+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:23.520 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:23.141+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:23.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:23.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:23.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:23.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:23.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:23.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:23.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:23.647+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:23.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:23.520+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:23.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:23.520+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:23.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:23.520+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:23.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:23.520+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:24.544 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:24.127+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:24.544 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:24.127+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:24.544 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:24.127+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:24.545 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:24.127+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:24.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:24.544+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:24.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:24.544+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:24.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:24.544+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:24.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:24.545+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:24.657+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:24.657+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:24.657+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:24.657+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:25.514 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:25.108+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:25.514 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:25.108+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:25.514 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:25.108+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:25.515 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:25.109+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:25.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:25.659+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:25.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:25.659+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:25.659+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:25.660+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:25.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:25.514+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:25.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:25.514+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:25.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:25.514+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:25.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:25.515+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:26.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:26.081+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:26.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:26.081+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:26.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:26.081+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:26.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:26.082+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:26.796 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:00:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:00:26.777+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 44702 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:00:26.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:26.648+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:26.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:26.648+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:26.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:26.648+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:26.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:26.649+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:26.797 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:26.542+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:26.797 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:26.542+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:26.798 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:26.542+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:26.798 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:26.543+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:27.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:27.075+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:27.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:27.075+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:27.075+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:27.076+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:27.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:27.516+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:27.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:27.516+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:27.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:27.516+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:27.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:27.518+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:27.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:27.637+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:27.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:27.637+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:27.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:27.637+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:27.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:27.638+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:28.514 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:28.115+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:28.515 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:28.115+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:28.515 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:28.115+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:28.515 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:28.115+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:28.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:28.514+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:28.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:28.514+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:28.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:28.514+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:28.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:28.516+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:28.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:28.599+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:28.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:28.599+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:28.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:28.599+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:28.600+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:29.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:29.079+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:29.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:29.079+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:29.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:29.079+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:29.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:29.080+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:29.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:29.520+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:29.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:29.520+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:29.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:29.520+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:29.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:29.521+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:29.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:29.624+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:29.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:29.624+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:29.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:29.624+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:29.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:29.625+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:30.539 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:30.104+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:30.539 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:30.104+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:30.539 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:30.104+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:30.539 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:30.105+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:30.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:30.539+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:30.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:30.539+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:30.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:30.539+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:30.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:30.541+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:30.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:30.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:30.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:30.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:30.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:30.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:30.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:30.661+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:31.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:31.064+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:31.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:31.064+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:31.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:31.064+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:31.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:31.065+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:31.795 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:00:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:00:31.782+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 44748 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:00:31.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:31.671+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:31.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:31.671+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:31.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:31.671+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:31.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:31.672+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:31.796 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:31.539+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:31.796 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:31.539+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:31.796 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:31.539+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:31.796 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:31.540+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:32.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:32.020+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:32.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:32.020+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:32.020+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:32.021+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:32.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:32.496+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:32.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:32.496+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:32.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:32.496+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:32.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:32.497+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:32.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:32.643+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:32.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:32.643+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:32.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:32.643+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:32.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:32.645+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:33.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:32.996+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:33.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:32.996+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:33.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:32.996+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:33.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:32.997+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:33.670 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:33.450+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:33.670 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:33.450+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:33.670 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:33.450+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:33.670 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:33.451+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:33.980 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:33.670+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:33.980 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:33.670+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:33.980 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:33.670+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:33.981 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:33.671+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:34.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:33.980+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:34.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:33.980+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:34.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:33.980+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:34.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:33.981+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:34.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:34.622+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:34.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:34.622+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:34.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:34.622+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:34.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:34.623+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:34.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:34.476+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:34.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:34.476+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:34.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:34.476+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:34.477+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:35.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:35.009+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:35.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:35.009+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:35.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:35.009+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:35.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:35.009+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:35.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:35.456+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:35.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:35.456+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:35.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:35.456+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:35.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:35.457+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:35.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:35.658+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:35.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:35.658+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:35.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:35.658+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:35.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:35.659+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:36.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:36.003+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:36.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:36.003+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:36.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:36.003+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:36.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:36.004+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:36.691 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:36.416+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:36.691 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:36.416+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:36.692 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:36.416+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:36.692 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:36.417+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:36.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:00:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:00:36.788+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 44793 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:00:36.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:36.691+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:36.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:36.691+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:36.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:36.691+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:36.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:36.692+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:37.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:36.972+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:37.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:36.972+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:37.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:36.972+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:37.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:36.973+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:37.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:37.459+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:37.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:37.459+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:37.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:37.459+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:37.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:37.459+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:37.664+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:37.664+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:37.664+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:37.665+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:38.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:37.932+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:38.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:37.932+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:37.932+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:37.933+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:38.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:38.639+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:38.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:38.639+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:38.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:38.639+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:38.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:38.640+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:38.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:38.477+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:38.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:38.477+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:38.477+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:38.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:38.478+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:39.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:38.920+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:39.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:38.920+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:39.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:38.920+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:39.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:38.920+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:39.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:39.454+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:39.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:39.454+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:39.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:39.454+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:39.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:39.455+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:39.966 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:39.685+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:39.966 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:39.685+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:39.966 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:39.685+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:39.966 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:39.686+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:40.260 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:39.966+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:40.260 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:39.966+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:40.260 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:39.966+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:40.260 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:39.967+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:40.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:40.419+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:40.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:40.419+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:40.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:40.419+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:40.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:40.420+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:40.979 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:40.667+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:40.979 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:40.667+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:40.979 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:40.667+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:40.979 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:40.668+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:41.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:40.979+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:41.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:40.979+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:41.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:40.979+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:41.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:40.980+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:41.794 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:41.658+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:41.794 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:41.658+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:41.794 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:41.658+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:41.794 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:41.658+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:41.795 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:41.436+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:41.795 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:41.436+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:41.795 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:41.436+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:41.795 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:41.437+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:42.087 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:00:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:00:41.794+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 44840 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:00:42.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:41.980+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:42.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:41.980+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:42.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:41.980+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:42.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:41.980+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:42.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:42.612+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:42.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:42.612+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:42.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:42.612+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:42.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:42.613+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:42.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:42.468+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:42.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:42.468+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:42.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:42.468+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:42.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:42.469+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:43.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:43.006+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:43.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:43.006+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:43.006+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:43.006+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:43.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:43.467+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:43.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:43.467+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:43.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:43.467+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:43.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:43.468+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:43.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:43.653+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:43.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:43.653+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:43.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:43.653+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:43.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:43.654+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:44.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:43.981+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:44.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:43.981+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:43.981+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:43.981+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:44.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:44.428+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:44.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:44.428+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:44.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:44.428+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:44.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:44.429+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:44.612+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:44.612+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:44.612+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:44.612+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:45.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:44.988+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:45.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:44.988+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:44.988+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:44.988+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:45.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:45.652+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:45.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:45.652+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:45.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:45.652+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:45.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:45.653+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:45.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:45.434+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:45.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:45.434+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:45.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:45.434+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:45.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:45.434+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:46.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:45.943+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:46.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:45.943+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:46.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:45.943+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:46.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:45.944+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:46.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:46.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:46.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:46.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:46.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:46.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:46.801 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:46.647+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:46.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:46.451+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:46.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:46.451+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:46.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:46.451+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:46.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:46.452+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:47.087 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:00:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:00:46.800+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 44886 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:00:47.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:46.946+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:47.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:46.946+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:47.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:46.946+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:47.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:46.946+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:47.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:47.614+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:47.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:47.614+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:47.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:47.614+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:47.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:47.615+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:47.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:47.465+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:47.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:47.465+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:47.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:47.465+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:47.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:47.466+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:48.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:47.928+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:48.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:47.928+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:48.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:47.928+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:48.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:47.928+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:48.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:48.635+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:48.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:48.635+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:48.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:48.635+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:48.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:48.636+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:48.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:48.441+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:48.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:48.441+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:48.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:48.441+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:48.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:48.442+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:49.244 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:48.962+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:49.244 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:48.962+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:49.244 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:48.962+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:49.244 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:48.962+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:49.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:49.475+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:49.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:49.475+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:49.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:49.475+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:49.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:49.476+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:49.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:49.665+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:49.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:49.665+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:49.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:49.665+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:49.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:49.666+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:50.261 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:49.948+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:50.261 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:49.948+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:50.261 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:49.948+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:50.261 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:49.949+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:50.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:50.441+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:50.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:50.441+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:50.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:50.441+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:50.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:50.442+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:50.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:50.647+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:50.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:50.647+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:50.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:50.647+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:50.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:50.648+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:51.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:50.971+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:51.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:50.971+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:51.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:50.971+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:51.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:50.972+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:51.799 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:51.607+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:51.799 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:51.607+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:51.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:51.607+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:51.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:51.608+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:51.800 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:51.475+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:51.800 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:51.475+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:51.800 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:51.475+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:51.800 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:51.476+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:52.087 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:00:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:00:51.805+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 44933 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:00:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:52.009+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:52.009+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:52.009+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:52.010+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:52.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:52.507+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:52.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:52.507+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:52.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:52.507+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:52.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:52.508+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:52.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:52.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:52.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:52.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:52.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:52.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:52.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:52.643+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:53.282 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:52.995+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:53.283 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:52.995+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:53.283 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:52.995+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:53.283 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:52.996+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:53.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:53.463+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:53.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:53.463+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:53.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:53.463+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:53.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:53.464+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:53.959 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:53.607+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:53.959 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:53.607+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:53.959 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:53.607+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:53.959 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:53.608+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:54.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:53.959+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:54.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:53.959+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:54.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:53.959+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:54.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:53.959+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:54.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:54.613+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:54.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:54.613+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:54.613+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:54.613+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:54.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:54.417+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:54.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:54.417+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:54.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:54.417+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:54.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:54.418+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:55.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:54.967+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:55.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:54.967+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:55.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:54.967+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:55.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:54.967+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:55.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:55.379+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:55.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:55.379+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:55.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:55.379+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:55.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:55.379+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:55.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:55.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:55.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:55.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:55.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:55.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:55.589+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:56.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:55.945+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:56.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:55.945+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:56.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:55.945+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:56.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:55.946+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:56.811 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:56.570+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:56.811 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:56.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:56.811 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:56.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:56.811 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:56.571+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:56.811 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:56.402+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:56.812 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:56.402+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:56.812 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:56.402+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:56.812 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:56.403+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:57.087 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:00:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:00:56.811+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 44978 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:00:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:56.943+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:56.943+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:56.943+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:56.944+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:57.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:57.590+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:57.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:57.590+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:57.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:57.590+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:57.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:57.591+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:57.427+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:57.427+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:57.427+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:57.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:57.428+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:58.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:57.988+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:58.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:57.988+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:57.988+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:57.989+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:58.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:58.628+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:58.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:58.628+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:58.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:58.628+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:00:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:58.629+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:58.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:58.450+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:58.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:58.450+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:58.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:58.450+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:58.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:58.451+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:00:59.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:58.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:59.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:58.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:59.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:58.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:00:59.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:00:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:00:58.958+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:00:59.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:59.461+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:59.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:59.461+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:59.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:59.461+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:00:59.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:00:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:00:59.462+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:00.006 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:59.600+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:00.007 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:59.600+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:00.007 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:59.600+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:00.007 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:00:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:00:59.601+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:00.261 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:00.007+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:00.262 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:00.007+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:00.262 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:00.007+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:00.262 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:00.008+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:00.554 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:00.509+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:00.555 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:00.509+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:00.555 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:00.509+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:00.555 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:00.510+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:00.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:00.555+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:00.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:00.555+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:00.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:00.555+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:00.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:00.555+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:01.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:00.992+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:01.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:00.992+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:00.992+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:00.993+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:01.799 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:01.593+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:01.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:01.593+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:01.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:01.593+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:01.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:01.594+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:01.800 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:01.526+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:01.800 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:01.526+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:01.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:01.526+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:01.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:01.527+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:02.087 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:01:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:01:01.816+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 45026 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:01:02.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:02.005+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:02.005+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:02.005+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:02.006+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:02.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:02.589+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:02.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:02.589+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:02.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:02.589+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:02.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:02.590+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:02.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:02.542+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:02.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:02.542+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:02.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:02.542+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:02.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:02.543+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:03.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:03.021+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:03.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:03.021+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:03.021+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:03.022+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:03.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:03.581+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:03.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:03.581+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:03.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:03.581+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:03.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:03.582+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:04.046 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:03.589+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:04.046 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:03.589+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:04.046 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:03.589+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:04.046 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:03.590+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:04.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:04.046+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:04.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:04.046+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:04.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:04.046+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:04.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:04.046+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:04.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:04.568+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:04.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:04.568+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:04.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:04.568+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:04.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:04.569+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:04.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:04.569+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:04.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:04.569+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:04.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:04.569+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:04.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:04.570+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:05.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:05.081+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:05.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:05.081+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:05.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:05.081+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:05.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:05.082+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:05.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:05.561+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:05.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:05.561+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:05.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:05.561+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:05.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:05.562+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:05.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:05.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:05.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:05.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:05.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:05.606+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:05.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:05.607+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:06.528 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:06.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:06.528 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:06.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:06.528 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:06.106+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:06.528 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:06.107+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:06.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:06.528+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:06.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:06.528+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:06.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:06.528+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:06.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:06.529+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:06.822 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:06.592+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:06.822 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:06.592+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:06.822 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:06.592+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:06.822 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:06.593+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:07.087 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:01:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:01:06.821+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 45072 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:01:07.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:07.063+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:07.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:07.063+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:07.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:07.063+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:07.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:07.064+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:07.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:07.559+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:07.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:07.559+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:07.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:07.559+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:07.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:07.560+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:07.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:07.567+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:07.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:07.567+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:07.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:07.567+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:07.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:07.567+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:08.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:08.055+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:08.055+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:08.055+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:08.055+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:08.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:08.596+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:08.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:08.597+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:08.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:08.597+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:08.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:08.598+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:08.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:08.554+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:08.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:08.554+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:08.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:08.554+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:08.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:08.555+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:09.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:09.089+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:09.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:09.089+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:09.524 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:09.089+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:09.524 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:09.090+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:09.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:09.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:09.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:09.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:09.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:09.595+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:09.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:09.596+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:09.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:09.523+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:09.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:09.523+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:09.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:09.523+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:09.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:09.524+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:10.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:10.074+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:10.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:10.074+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:10.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:10.074+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:10.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:10.075+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:10.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:10.551+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:10.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:10.551+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:10.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:10.551+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:10.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:10.552+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:10.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:10.544+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:10.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:10.544+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:10.544+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:10.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:10.546+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:11.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:11.067+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:11.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:11.067+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:11.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:11.067+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:11.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:11.068+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:11.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:11.543+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:11.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:11.543+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:11.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:11.543+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:11.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:11.544+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:11.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:11.590+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:11.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:11.590+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:11.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:11.590+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:11.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:11.590+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:12.087 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:01:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:01:11.827+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 45119 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:01:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:12.022+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:12.022+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:12.022+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:12.023+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:12.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:12.574+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:12.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:12.574+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:12.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:12.574+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:12.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:12.575+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:12.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:12.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:12.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:12.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:12.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:12.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:12.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:12.617+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:13.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:13.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:13.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:13.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:13.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:13.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:13.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:13.052+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:13.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:13.534+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:13.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:13.534+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:13.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:13.534+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:13.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:13.534+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:13.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:13.597+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:13.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:13.597+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:13.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:13.597+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:13.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:13.597+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:14.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:14.075+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:14.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:14.075+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:14.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:14.075+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:14.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:14.075+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:14.787 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:14.599+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:14.787 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:14.599+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:14.787 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:14.599+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:14.787 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:14.600+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:14.788 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:14.488+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:14.788 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:14.490+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:14.788 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:14.490+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:14.788 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:14.490+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:15.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:15.065+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:15.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:15.065+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:15.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:15.065+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:15.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:15.066+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:15.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:15.461+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:15.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:15.461+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:15.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:15.461+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:15.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:15.462+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:15.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:15.639+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:15.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:15.639+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:15.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:15.639+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:15.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:15.640+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:16.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:16.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:16.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:16.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:16.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:16.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:16.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:16.058+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:16.832 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:16.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:16.832 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:16.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:16.832 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:16.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:16.832 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:16.480+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:16.833 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:16.678+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:16.833 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:16.678+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:16.833 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:16.678+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:16.833 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:16.679+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:17.087 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:01:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:01:16.833+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 45164 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:01:17.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:17.009+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:17.009+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:17.009+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:17.009+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:17.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:17.656+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:17.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:17.656+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:17.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:17.656+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:17.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:17.657+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:17.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:17.460+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:17.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:17.460+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:17.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:17.460+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:17.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:17.461+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:18.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:17.984+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:18.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:17.984+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:17.984+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:17.985+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:18.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:18.473+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:18.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:18.473+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:18.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:18.473+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:18.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:18.474+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:18.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:18.671+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:18.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:18.671+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:18.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:18.671+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:18.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:18.672+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:19.244 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:18.952+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:19.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:18.952+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:19.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:18.952+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:19.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:18.953+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:19.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:19.424+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:19.424+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:19.424+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:19.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:19.425+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:20.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:19.697+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:20.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:19.697+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:20.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:19.697+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:20.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:19.698+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:20.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:19.918+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:20.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:19.918+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:20.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:19.918+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:20.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:19.919+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:20.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:20.417+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:20.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:20.417+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:20.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:20.417+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:20.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:20.418+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:21.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:20.741+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:21.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:20.741+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:21.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:20.741+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:21.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:20.742+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:21.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:20.950+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:21.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:20.950+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:21.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:20.950+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:21.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:20.950+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:21.775 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:21.421+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:21.775 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:21.421+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:21.775 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:21.421+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:21.776 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:21.422+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:22.087 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:01:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:01:21.839+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 45211 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:01:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:21.935+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:21.935+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:21.935+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:21.936+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:21.776+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:21.776+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:22.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:21.776+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:22.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:21.776+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:22.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:22.385+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:22.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:22.385+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:22.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:22.385+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:22.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:22.386+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:23.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:22.924+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:23.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:22.924+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:22.924+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:22.924+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:22.740+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:22.740+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:22.740+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:22.741+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:23.716 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:23.430+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:23.716 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:23.430+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:23.717 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:23.430+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:23.717 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:23.431+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:24.003 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:23.716+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:24.003 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:23.716+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:24.003 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:23.716+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:24.003 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:23.717+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:24.003 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:23.907+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:24.004 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:23.907+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:24.004 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:23.907+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:24.004 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:23.908+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:24.747 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:24.384+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:24.748 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:24.384+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:24.748 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:24.384+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:24.748 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:24.385+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:25.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:24.747+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:25.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:24.747+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:25.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:24.747+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:25.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:24.748+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:24.865+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:24.865+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:24.865+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:24.865+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:25.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:25.367+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:25.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:25.367+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:25.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:25.367+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:25.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:25.368+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:26.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:25.724+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:26.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:25.724+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:26.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:25.724+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:26.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:25.725+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:26.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:25.893+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:26.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:25.893+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:26.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:25.893+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:26.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:25.894+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:26.713 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:26.398+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:26.713 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:26.398+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:26.713 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:26.398+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:26.713 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:26.399+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:26.964 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:01:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:01:26.843+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 45257 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:01:26.964 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:26.713+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:26.965 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:26.713+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:26.965 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:26.713+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:26.965 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:26.714+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:26.965 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:26.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:26.965 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:26.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:26.965 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:26.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:26.966 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:26.930+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:27.750 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:27.360+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:27.751 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:27.360+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:27.751 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:27.360+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:27.751 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:27.362+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:28.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:27.750+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:27.750+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:27.750+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:28.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:27.751+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:28.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:27.903+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:28.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:27.903+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:27.903+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:27.904+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:28.709 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:28.366+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:28.710 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:28.366+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:28.710 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:28.366+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:28.710 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:28.368+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:29.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:28.943+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:29.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:28.943+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:28.943+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:28.943+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:29.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:28.709+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:29.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:28.709+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:29.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:28.709+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:29.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:28.710+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:29.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:29.364+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:29.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:29.364+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:29.364+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:29.365+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:30.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:29.747+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:30.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:29.747+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:30.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:29.747+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:30.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:29.748+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:29.932+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:29.932+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:29.932+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:30.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:29.932+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:30.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:30.334+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:30.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:30.334+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:30.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:30.334+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:30.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:30.335+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:31.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:30.931+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:31.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:30.931+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:31.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:30.931+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:31.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:30.931+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:31.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:30.699+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:31.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:30.699+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:30.699+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:31.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:30.699+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:31.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:31.329+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:31.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:31.329+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:31.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:31.329+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:31.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:31.330+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:31.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:01:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:01:31.849+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 45300 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:01:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:31.707+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:31.707+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:31.707+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:31.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:31.708+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:31.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:31.950+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:31.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:31.950+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:31.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:31.950+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:31.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:31.951+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:32.727 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:32.358+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:32.727 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:32.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:32.727 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:32.359+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:32.727 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:32.360+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:33.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:32.727+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:33.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:32.727+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:33.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:32.727+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:33.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:32.727+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:32.938+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:32.938+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:32.938+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:33.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:32.938+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:33.742 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:33.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:33.742 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:33.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:33.742 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:33.405+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:33.742 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:33.406+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:34.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:33.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:34.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:33.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:34.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:33.742+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:34.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:33.743+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:33.986+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:33.986+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:33.986+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:33.987+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:34.727 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:34.360+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:34.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:34.360+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:34.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:34.360+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:34.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:34.361+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:35.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:34.939+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:35.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:34.939+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:34.939+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:34.940+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:35.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:34.727+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:35.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:34.727+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:35.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:34.727+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:35.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:34.728+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:35.776 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:35.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:35.776 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:35.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:35.776 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:35.401+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:35.776 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:35.402+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:36.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:35.891+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:36.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:35.891+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:36.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:35.891+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:36.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:35.892+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:36.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:35.776+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:36.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:35.776+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:35.776+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:35.777+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:36.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:36.431+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:36.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:36.431+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:36.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:36.431+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:36.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:36.432+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:37.087 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:01:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:01:36.855+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 45345 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:01:37.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:36.800+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:37.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:36.800+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:37.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:36.800+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:37.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:36.801+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:36.908+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:36.908+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:36.908+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:36.909+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:37.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:37.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:37.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:37.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:37.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:37.449+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:37.450+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:38.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:37.849+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:38.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:37.849+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:38.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:37.849+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:38.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:37.849+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:37.863+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:37.863+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:37.863+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:38.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:37.863+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:38.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:38.499+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:38.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:38.499+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:38.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:38.499+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:38.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:38.500+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:39.244 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:38.887+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:39.244 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:38.887+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:39.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:38.887+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:39.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:38.887+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:39.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:38.845+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:39.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:38.845+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:39.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:38.845+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:39.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:38.846+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:39.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:39.538+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:39.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:39.538+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:39.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:39.538+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:39.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:39.540+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:40.265 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:39.877+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:40.265 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:39.877+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:40.265 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:39.877+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:40.265 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:39.879+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:40.265 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:39.864+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:40.265 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:39.864+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:40.266 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:39.864+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:40.266 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:39.865+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:40.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:40.580+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:40.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:40.580+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:40.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:40.580+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:40.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:40.581+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:41.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:40.850+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:41.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:40.850+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:41.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:40.850+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:41.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:40.851+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:41.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:40.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:41.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:40.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:41.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:40.848+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:41.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:40.848+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:41.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:41.566+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:41.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:41.566+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:41.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:41.566+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:41.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:41.566+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:42.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:01:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:01:41.860+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 45392 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:01:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:41.878+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:41.878+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:41.878+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:41.878+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:42.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:41.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:42.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:41.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:42.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:41.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:42.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:41.861+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:42.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:42.540+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:42.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:42.540+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:42.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:42.540+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:42.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:42.541+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:43.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:42.891+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:43.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:42.891+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:43.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:42.891+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:43.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:42.892+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:42.888+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:42.888+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:42.888+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:42.888+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:43.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:43.551+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:43.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:43.551+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:43.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:43.551+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:43.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:43.552+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:44.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:43.935+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:44.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:43.935+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:43.935+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:43.936+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:43.903+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:43.903+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:43.903+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:44.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:43.903+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:44.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:44.526+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:44.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:44.526+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:44.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:44.526+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:44.526+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:45.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:44.912+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:45.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:44.912+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:45.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:44.912+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:45.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:44.913+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:44.863+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:44.863+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:44.863+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:44.864+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:45.830 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:45.522+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:45.830 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:45.522+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:45.831 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:45.522+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:45.831 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:45.523+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:46.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:45.886+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:46.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:45.886+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:46.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:45.886+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:46.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:45.886+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:46.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:45.830+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:46.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:45.830+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:46.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:45.830+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:46.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:45.831+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:46.799 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:46.493+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:46.799 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:46.493+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:46.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:46.493+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:46.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:46.494+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:47.087 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:01:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:01:46.866+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 45438 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:01:47.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:46.928+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:47.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:46.928+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:47.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:46.928+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:47.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:46.929+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:46.799+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:46.799+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:46.799+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:47.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:46.800+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:47.829 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:47.482+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:47.829 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:47.482+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:47.830 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:47.482+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:47.830 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:47.483+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:48.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:47.929+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:48.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:47.929+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:48.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:47.929+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:48.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:47.930+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:48.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:47.829+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:48.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:47.829+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:48.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:47.829+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:48.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:47.830+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:48.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:48.465+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:48.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:48.465+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:48.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:48.465+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:48.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:48.466+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:49.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:48.800+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:49.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:48.800+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:49.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:48.800+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:49.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:48.800+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:48.908+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:48.908+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:48.908+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:49.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:48.909+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:49.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:49.430+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:49.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:49.430+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:49.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:49.430+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:49.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:49.431+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:50.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:49.887+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:49.887+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:49.887+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:50.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:49.888+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:49.790+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:49.790+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:49.790+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:50.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:49.791+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:50.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:50.451+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:50.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:50.451+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:50.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:50.451+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:50.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:50.452+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:51.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:50.861+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:51.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:50.861+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:51.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:50.861+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:51.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:50.862+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:51.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:50.754+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:51.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:50.754+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:51.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:50.754+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:51.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:50.755+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:51.729 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:51.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:51.730 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:51.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:51.730 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:51.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:51.730 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:51.480+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:51.993 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:01:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:01:51.872+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 45485 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:01:51.993 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:51.821+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:51.993 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:51.821+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:51.993 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:51.821+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:51.993 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:51.822+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:51.994 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:51.730+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:51.994 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:51.730+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:51.994 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:51.730+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:51.994 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:51.731+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:52.743 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:52.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:52.744 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:52.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:52.744 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:52.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:52.744 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:52.480+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:53.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:52.744+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:52.744+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:52.744+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:52.745+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:53.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:52.806+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:53.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:52.806+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:53.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:52.806+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:53.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:52.807+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:53.808 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:53.518+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:53.809 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:53.518+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:53.809 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:53.518+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:53.809 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:53.518+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:53.809 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:53.699+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:53.809 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:53.699+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:53.809 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:53.699+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:53.810 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:53.700+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:54.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:53.809+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:54.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:53.809+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:54.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:53.809+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:54.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:53.810+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:54.831 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:54.540+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:54.832 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:54.540+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:54.832 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:54.540+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:54.832 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:54.541+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:54.832 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:54.723+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:54.832 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:54.723+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:54.833 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:54.723+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:54.833 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:54.724+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:55.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:54.832+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:55.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:54.832+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:55.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:54.832+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:55.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:54.833+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:55.828 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:55.532+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:55.828 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:55.532+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:55.829 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:55.532+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:55.829 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:55.532+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:55.829 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:55.714+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:55.829 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:55.714+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:55.829 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:55.714+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:55.829 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:55.715+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:56.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:55.828+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:56.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:55.828+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:56.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:55.828+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:56.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:55.829+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:56.815 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:56.498+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:56.815 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:56.498+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:56.815 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:56.498+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:56.816 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:56.499+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:56.816 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:56.719+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:56.816 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:56.719+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:56.816 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:56.719+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:56.817 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:56.720+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:57.087 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:01:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:01:56.877+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 45530 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:01:57.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:56.815+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:57.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:56.815+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:57.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:56.815+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:57.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:56.816+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:57.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:57.503+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:57.801 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:57.503+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:57.801 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:57.503+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:57.801 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:57.504+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:57.801 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:57.729+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:57.801 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:57.729+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:57.802 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:57.729+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:57.802 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:57.730+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:58.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:57.801+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:58.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:57.801+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:58.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:57.801+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:58.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:57.802+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:01:58.805 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:58.547+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:58.806 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:58.547+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:58.806 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:58.547+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:01:58.806 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:58.548+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:58.806 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:58.772+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:58.806 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:58.773+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:58.806 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:58.773+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:01:58.807 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:58.773+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:01:59.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:58.806+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:59.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:58.806+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:59.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:58.806+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:01:59.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:58.807+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:00.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:59.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:00.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:59.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:00.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:59.762+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:00.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:01:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:01:59.763+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:00.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:59.591+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:00.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:59.591+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:59.591+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:00.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:01:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:01:59.592+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:59.821+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:59.821+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:59.821+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:01:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:01:59.822+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:01.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:00.786+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:01.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:00.786+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:01.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:00.786+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:01.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:00.788+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:01.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:00.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:01.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:00.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:01.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:00.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:01.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:00.643+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:00.851+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:00.851+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:00.851+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:00.852+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:01.894 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:02:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:02:01.883+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 45577 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:02:01.894 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:01.787+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:01.895 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:01.787+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:01.895 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:01.787+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:01.895 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:01.788+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:01.895 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:01.643+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:01.895 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:01.643+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:01.895 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:01.643+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:01.896 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:01.644+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:02.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:01.894+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:02.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:01.894+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:02.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:01.894+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:02.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:01.895+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:03.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:02.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:03.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:02.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:03.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:02.744+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:03.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:02.745+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:03.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:02.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:03.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:02.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:03.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:02.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:03.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:02.906+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:03.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:02.664+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:03.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:02.664+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:03.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:02.664+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:03.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:02.665+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:03.939 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:03.679+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:03.939 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:03.679+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:03.939 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:03.679+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:03.939 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:03.679+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:03.939 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:03.765+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:03.940 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:03.765+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:03.940 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:03.765+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:03.940 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:03.766+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:03.940 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:03.888+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:03.940 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:03.888+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:03.940 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:03.888+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:03.940 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:03.889+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:05.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:04.712+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:05.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:04.712+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:05.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:04.712+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:05.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:04.713+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:05.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:04.855+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:05.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:04.855+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:05.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:04.855+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:05.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:04.855+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:04.767+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:04.767+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:04.767+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:05.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:04.768+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:06.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:05.753+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:06.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:05.753+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:06.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:05.753+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:06.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:05.753+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:06.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:05.714+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:06.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:05.714+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:06.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:05.714+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:06.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:05.715+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:06.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:05.809+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:06.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:05.810+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:06.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:05.810+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:06.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:05.810+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:07.010 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:02:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:02:06.889+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 45623 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:02:07.010 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:06.729+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:07.011 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:06.729+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:07.011 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:06.729+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:07.011 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:06.730+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:07.011 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:06.803+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:07.011 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:06.803+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:07.012 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:06.803+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:07.012 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:06.803+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:07.012 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:06.680+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:07.012 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:06.681+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:07.013 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:06.681+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:07.013 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:06.681+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:08.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:07.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:08.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:07.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:08.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:07.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:08.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:07.642+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:08.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:07.840+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:08.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:07.840+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:07.840+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:08.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:07.841+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:07.746+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:07.746+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:07.746+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:08.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:07.747+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:09.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:08.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:09.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:08.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:09.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:08.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:09.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:08.857+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:09.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:08.643+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:09.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:08.643+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:09.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:08.644+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:09.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:08.644+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:08.709+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:09.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:08.709+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:09.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:08.709+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:09.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:08.710+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:10.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:09.687+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:10.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:09.687+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:10.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:09.687+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:10.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:09.687+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:10.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:09.747+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:10.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:09.747+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:10.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:09.747+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:10.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:09.748+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:09.809+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:09.809+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:09.809+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:10.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:09.810+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:11.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:10.720+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:11.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:10.720+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:11.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:10.720+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:11.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:10.720+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:10.789+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:10.789+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:10.789+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:11.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:10.790+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:10.843+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:10.843+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:10.843+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:10.844+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:12.016 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:02:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:02:11.895+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 45670 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:02:12.017 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:11.862+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:12.017 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:11.862+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:12.017 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:11.862+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:12.017 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:11.863+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:12.018 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:11.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:12.018 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:11.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:12.018 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:11.705+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:12.018 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:11.706+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:12.018 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:11.746+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:12.019 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:11.746+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:12.019 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:11.746+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:12.019 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:11.747+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:13.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:12.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:12.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:12.857+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:12.857+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:13.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:12.682+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:13.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:12.682+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:12.682+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:13.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:12.683+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:12.704+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:12.704+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:13.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:12.704+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:13.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:12.705+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:14.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:13.658+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:14.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:13.658+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:13.658+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:14.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:13.658+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:14.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:13.699+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:14.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:13.699+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:14.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:13.699+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:14.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:13.700+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:13.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:13.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:13.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:13.858+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:14.899 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:14.678+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:14.899 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:14.678+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:14.900 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:14.678+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:14.900 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:14.679+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:14.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:14.609+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:14.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:14.609+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:14.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:14.609+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:14.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:14.609+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:15.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:14.899+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:15.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:14.899+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:15.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:14.899+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:15.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:14.900+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:16.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:15.862+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:16.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:15.862+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:16.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:15.862+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:16.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:15.863+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:16.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:15.720+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:16.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:15.720+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:16.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:15.720+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:16.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:15.722+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:15.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:15.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:15.627+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:16.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:15.628+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:16.860 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:16.717+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:16.860 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:16.717+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:16.861 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:16.717+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:16.861 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:16.718+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:16.861 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:16.609+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:16.861 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:16.609+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:16.861 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:16.609+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:16.862 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:16.610+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:17.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:02:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:02:16.901+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 45715 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:02:17.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:16.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:16.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:16.860+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:16.861+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:17.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:17.713+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:17.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:17.713+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:17.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:17.713+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:17.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:17.713+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:17.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:17.567+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:17.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:17.567+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:17.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:17.567+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:17.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:17.568+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:18.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:17.847+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:18.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:17.847+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:18.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:17.847+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:17.847+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:18.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:18.731+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:18.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:18.731+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:18.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:18.731+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:18.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:18.732+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:18.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:18.561+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:18.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:18.561+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:18.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:18.561+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:18.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:18.562+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:19.244 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:18.897+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:19.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:18.897+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:19.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:18.897+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:19.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:18.898+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:19.879 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:19.766+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:19.879 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:19.766+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:19.879 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:19.766+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:19.879 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:19.767+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:19.880 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:19.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:19.880 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:19.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:19.880 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:19.603+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:19.880 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:19.604+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:20.268 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:19.879+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:20.268 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:19.879+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:20.268 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:19.879+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:20.268 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:19.880+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:20.918 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:20.816+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:20.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:20.816+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:20.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:20.816+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:20.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:20.817+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:20.919 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:20.642+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:20.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:20.642+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:20.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:20.642+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:20.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:20.643+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:21.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:20.919+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:21.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:20.919+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:21.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:20.919+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:21.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:20.919+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:21.882 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:21.857+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:21.882 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:21.857+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:21.882 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:21.857+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:21.882 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:21.858+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:21.883 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:21.621+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:21.883 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:21.621+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:21.883 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:21.621+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:21.883 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:21.622+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:22.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:02:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:02:21.906+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 45762 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:02:22.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:21.882+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:22.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:21.882+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:22.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:21.882+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:22.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:21.883+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:23.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:22.886+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:23.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:22.886+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:22.886+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:22.887+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:23.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:22.833+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:23.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:22.833+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:23.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:22.833+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:23.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:22.834+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:22.662+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:22.662+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:22.662+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:23.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:22.662+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:24.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:23.712+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:24.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:23.712+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:24.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:23.712+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:24.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:23.713+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:23.894+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:23.894+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:23.894+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:23.894+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:24.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:23.827+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:24.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:23.827+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:24.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:23.827+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:24.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:23.828+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:25.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:24.799+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:25.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:24.799+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:25.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:24.799+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:25.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:24.800+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:25.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:24.738+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:25.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:24.738+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:25.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:24.738+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:25.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:24.739+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:24.927+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:24.927+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:24.927+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:25.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:24.927+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:26.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:25.794+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:26.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:25.794+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:26.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:25.794+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:26.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:25.795+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:26.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:25.753+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:26.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:25.753+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:26.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:25.753+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:26.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:25.753+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:25.969+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:25.969+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:25.969+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:26.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:25.969+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:26.989 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:02:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:02:26.913+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 45808 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:02:26.990 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:26.737+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:26.990 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:26.737+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:26.990 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:26.737+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:26.990 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:26.737+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:26.990 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:26.760+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:26.990 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:26.760+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:26.991 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:26.760+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:26.991 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:26.761+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:27.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:26.989+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:27.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:26.989+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:26.989+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:26.990+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:27.981 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:27.767+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:27.981 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:27.767+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:27.981 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:27.767+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:27.981 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:27.768+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:27.981 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:27.721+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:27.982 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:27.721+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:27.982 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:27.721+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:27.982 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:27.722+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:28.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:27.981+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:28.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:27.981+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:28.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:27.981+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:28.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:27.982+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:29.009 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:28.744+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:29.010 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:28.744+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:29.010 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:28.744+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:29.010 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:28.745+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:29.010 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:28.792+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:29.010 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:28.792+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:29.010 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:28.792+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:29.011 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:28.793+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:29.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:29.009+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:29.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:29.009+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:29.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:29.009+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:29.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:29.010+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:30.039 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:29.781+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:30.039 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:29.781+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:30.039 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:29.781+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:30.039 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:29.782+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:30.040 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:29.801+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:30.040 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:29.801+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:30.040 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:29.801+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:30.040 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:29.801+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:30.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:30.039+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:30.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:30.039+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:30.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:30.039+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:30.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:30.040+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:31.065 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:30.736+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:31.065 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:30.736+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:31.066 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:30.736+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:31.066 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:30.736+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:31.066 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:30.777+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:31.066 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:30.777+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:31.066 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:30.777+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:31.066 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:30.778+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:31.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:31.065+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:31.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:31.065+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:31.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:31.065+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:31.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:31.066+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:31.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:31.781+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:31.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:31.781+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:31.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:31.781+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:31.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:31.781+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:31.919 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:31.774+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:31.919 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:31.774+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:31.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:31.774+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:31.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:31.775+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:32.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:02:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:02:31.919+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 45855 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:02:32.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:32.099+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:32.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:32.099+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:32.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:32.099+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:32.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:32.099+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:33.059 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:32.799+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:33.059 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:32.799+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:33.059 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:32.799+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:33.060 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:32.799+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:33.060 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:32.766+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:33.060 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:32.766+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:33.060 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:32.766+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:33.060 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:32.767+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:33.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:33.059+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:33.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:33.059+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:33.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:33.059+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:33.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:33.060+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:34.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:33.803+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:34.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:33.803+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:34.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:33.803+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:34.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:33.803+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:34.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:33.819+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:34.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:33.819+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:34.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:33.819+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:34.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:33.820+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:34.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:34.107+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:34.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:34.107+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:34.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:34.107+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:34.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:34.107+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:35.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:34.832+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:35.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:34.832+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:35.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:34.832+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:35.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:34.832+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:35.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:34.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:35.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:34.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:35.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:34.866+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:35.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:34.867+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:35.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:35.096+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:35.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:35.096+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:35.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:35.096+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:35.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:35.096+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:36.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:35.824+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:36.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:35.824+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:36.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:35.824+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:36.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:35.825+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:36.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:35.833+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:36.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:35.833+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:35.833+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:36.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:35.835+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:36.050+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:36.050+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:36.050+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:36.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:36.051+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:37.056 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:02:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:02:36.925+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 45900 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:02:37.056 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:36.862+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:37.057 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:36.862+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:37.057 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:36.862+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:37.057 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:36.863+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:37.057 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:36.799+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:37.057 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:36.799+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:37.057 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:36.799+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:37.057 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:36.800+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:37.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:37.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:37.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:37.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:37.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:37.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:37.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:37.058+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:38.075 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:37.759+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:38.076 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:37.759+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:38.076 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:37.759+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:38.076 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:37.760+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:38.076 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:37.909+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:38.076 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:37.909+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:38.077 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:37.909+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:38.077 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:37.910+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:38.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:38.077+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:38.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:38.077+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:38.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:38.077+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:38.077+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:39.057 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:38.763+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:39.057 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:38.763+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:39.058 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:38.763+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:39.058 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:38.764+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:39.058 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:38.932+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:39.058 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:38.932+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:39.058 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:38.932+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:39.058 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:38.932+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:39.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:39.058+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:39.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:39.058+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:39.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:39.058+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:39.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:39.059+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:40.075 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:39.768+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:40.076 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:39.768+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:40.076 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:39.768+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:40.076 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:39.769+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:40.076 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:39.925+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:40.077 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:39.925+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:40.077 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:39.925+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:40.077 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:39.926+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:40.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:40.077+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:40.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:40.077+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:40.078 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:40.077+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:40.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:40.077+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:41.076 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:40.722+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:41.076 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:40.722+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:41.076 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:40.722+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:41.076 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:40.723+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:41.077 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:40.970+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:41.077 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:40.970+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:41.077 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:40.970+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:41.077 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:40.970+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:41.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:41.077+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:41.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:41.077+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:41.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:41.077+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:41.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:41.078+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:42.040 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:02:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:02:41.931+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 45947 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:02:42.040 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:41.736+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:42.040 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:41.736+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:42.040 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:41.736+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:42.041 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:41.736+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:42.041 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:41.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:42.041 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:41.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:42.041 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:41.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:42.041 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:41.958+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:42.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:42.041+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:42.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:42.041+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:42.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:42.041+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:42.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:42.042+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:42.988 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:42.739+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:42.989 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:42.739+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:42.989 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:42.739+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:42.989 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:42.739+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:43.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:42.989+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:43.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:42.989+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:43.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:42.989+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:43.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:42.990+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:43.084+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:43.084+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:43.084+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:43.085+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:43.955 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:43.701+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:43.955 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:43.701+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:43.955 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:43.701+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:43.956 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:43.701+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:44.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:43.955+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:44.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:43.955+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:43.955+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:43.956+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:44.082+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:44.082+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:44.082+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:44.083+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:44.969 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:44.668+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:44.969 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:44.668+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:44.970 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:44.668+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:44.970 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:44.669+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:45.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:44.969+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:45.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:44.969+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:45.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:44.969+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:45.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:44.970+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:45.123+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:45.123+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:45.123+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:45.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:45.124+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:46.018 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:45.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:46.019 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:45.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:46.019 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:45.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:46.019 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:45.643+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:46.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:46.018+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:46.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:46.018+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:46.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:46.018+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:46.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:46.019+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:46.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:46.117+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:46.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:46.117+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:46.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:46.117+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:46.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:46.117+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:46.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:46.644+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:46.936 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:46.644+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:46.936 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:46.644+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:46.936 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:46.644+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:47.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:02:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:02:46.936+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 45993 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:02:47.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:47.148+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:47.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:47.148+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:47.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:47.148+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:47.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:47.149+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:47.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:47.028+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:47.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:47.028+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:47.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:47.028+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:47.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:47.029+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:48.013 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:47.638+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:48.013 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:47.638+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:48.013 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:47.638+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:48.013 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:47.638+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:48.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:48.013+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:48.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:48.013+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:48.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:48.013+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:48.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:48.014+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:48.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:48.127+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:48.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:48.127+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:48.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:48.127+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:48.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:48.127+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:49.047 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:48.594+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:49.048 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:48.594+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:49.048 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:48.594+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:49.048 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:48.595+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:49.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:49.048+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:49.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:49.048+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:49.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:49.048+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:49.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:49.048+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:49.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:49.092+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:49.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:49.092+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:49.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:49.092+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:49.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:49.092+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:49.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:49.563+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:49.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:49.564+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:49.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:49.564+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:49.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:49.564+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:50.270 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:50.000+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:50.270 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:50.000+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:50.270 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:50.000+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:50.270 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:50.001+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:50.270 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:50.047+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:50.271 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:50.047+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:50.271 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:50.047+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:50.271 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:50.048+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:50.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:50.566+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:50.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:50.566+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:50.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:50.566+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:50.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:50.567+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:51.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:50.996+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:51.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:50.996+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:51.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:50.996+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:51.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:50.996+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:51.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:51.006+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:51.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:51.006+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:51.006+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:51.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:51.007+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:51.805 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:51.546+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:51.806 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:51.546+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:51.806 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:51.546+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:51.806 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:51.547+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:52.062 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:02:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:02:51.941+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 46040 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:02:52.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:51.988+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:52.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:51.988+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:52.063 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:51.988+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:52.063 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:51.988+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:52.063 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:51.971+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:52.063 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:51.971+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:52.063 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:51.971+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:52.063 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:51.972+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:52.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:52.514+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:52.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:52.514+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:52.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:52.514+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:52.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:52.514+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:53.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:52.975+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:53.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:52.975+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:53.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:52.975+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:53.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:52.976+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:53.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:53.016+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:53.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:53.016+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:53.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:53.016+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:53.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:53.017+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:53.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:53.518+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:53.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:53.518+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:53.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:53.518+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:53.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:53.519+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:54.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:53.990+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:54.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:53.990+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:54.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:53.990+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:54.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:53.991+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:54.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:53.970+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:54.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:53.970+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:54.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:53.970+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:54.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:53.970+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:54.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:54.486+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:54.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:54.486+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:54.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:54.486+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:54.487+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:55.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:54.959+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:55.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:54.959+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:55.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:54.959+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:55.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:54.960+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:55.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:54.931+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:55.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:54.931+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:55.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:54.931+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:55.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:54.932+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:55.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:55.508+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:55.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:55.508+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:55.508+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:55.508+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:56.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:55.905+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:56.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:55.905+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:56.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:55.905+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:56.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:55.906+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:56.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:55.981+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:56.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:55.981+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:56.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:55.981+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:56.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:55.982+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:56.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:56.543+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:56.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:56.543+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:56.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:56.543+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:56.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:56.544+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:57.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:02:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:02:56.947+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 46085 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:02:57.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:56.940+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:57.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:56.940+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:57.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:56.940+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:57.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:56.941+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:57.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:56.868+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:57.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:56.868+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:56.868+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:57.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:56.869+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:57.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:57.544+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:57.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:57.544+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:57.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:57.544+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:57.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:57.545+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:58.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:57.904+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:58.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:57.904+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:58.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:57.904+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:58.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:57.905+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:57.910+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:58.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:57.910+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:58.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:57.910+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:58.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:57.911+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:58.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:58.570+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:58.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:58.570+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:58.570+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:58.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:58.570+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:59.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:58.880+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:59.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:58.880+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:59.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:58.880+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:02:59.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:58.881+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:02:59.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:58.889+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:59.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:58.889+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:59.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:58.889+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:02:59.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:58.890+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:02:59.873 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:59.604+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:59.873 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:59.604+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:59.873 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:59.604+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:02:59.873 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:02:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:02:59.605+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:00.270 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:59.925+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:00.270 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:59.925+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:00.271 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:59.925+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:00.271 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:02:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:02:59.926+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:00.271 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:59.873+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:00.271 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:59.873+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:00.271 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:59.873+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:00.271 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:02:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:02:59.874+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:00.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:00.581+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:00.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:00.581+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:00.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:00.581+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:00.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:00.582+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:01.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:00.912+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:01.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:00.912+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:01.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:00.912+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:01.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:00.914+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:00.900+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:00.900+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:00.900+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:00.901+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:01.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:01.557+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:01.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:01.557+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:01.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:01.557+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:01.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:01.557+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:02.306 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:03:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:03:01.953+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 46132 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:03:02.307 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:01.877+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:02.307 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:01.877+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:02.307 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:01.877+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:02.307 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:01.878+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:02.308 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:01.894+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:02.308 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:01.894+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:02.308 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:01.894+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:02.308 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:01.895+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:02.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:02.538+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:02.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:02.538+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:02.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:02.538+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:02.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:02.539+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:03.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:02.911+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:03.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:02.911+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:03.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:02.911+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:03.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:02.911+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:02.878+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:02.878+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:02.878+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:02.879+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:03.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:03.512+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:03.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:03.512+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:03.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:03.512+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:03.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:03.512+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:04.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:03.915+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:04.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:03.915+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:04.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:03.915+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:04.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:03.916+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:04.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:03.861+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:04.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:03.861+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:04.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:03.861+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:04.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:03.862+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:04.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:04.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:04.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:04.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:04.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:04.479+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:04.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:04.480+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:05.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:04.944+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:05.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:04.944+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:05.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:04.944+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:05.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:04.945+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:05.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:04.911+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:05.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:04.911+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:05.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:04.911+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:05.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:04.912+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:05.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:05.517+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:05.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:05.517+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:05.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:05.517+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:05.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:05.518+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:06.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:05.915+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:06.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:05.915+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:06.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:05.915+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:06.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:05.916+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:06.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:05.865+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:06.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:05.865+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:06.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:05.865+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:06.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:05.866+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:06.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:06.476+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:06.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:06.476+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:06.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:06.476+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:06.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:06.476+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:07.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:03:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:03:06.959+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 46178 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:03:07.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:06.958+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:07.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:06.958+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:07.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:06.958+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:07.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:06.959+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:07.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:06.850+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:07.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:06.850+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:07.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:06.850+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:07.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:06.850+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:07.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:07.515+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:07.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:07.515+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:07.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:07.515+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:07.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:07.516+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:08.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:07.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:08.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:07.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:08.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:07.957+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:08.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:07.958+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:07.878+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:07.878+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:07.878+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:08.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:07.879+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:08.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:08.480+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:08.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:08.480+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:08.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:08.480+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:08.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:08.481+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:09.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:08.916+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:09.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:08.916+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:09.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:08.916+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:09.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:08.917+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:09.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:08.867+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:09.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:08.867+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:09.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:08.867+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:09.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:08.867+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:09.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:09.468+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:09.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:09.468+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:09.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:09.468+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:09.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:09.469+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:10.271 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:09.906+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:10.271 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:09.907+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:10.271 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:09.907+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:10.271 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:09.907+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:10.272 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:09.904+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:10.272 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:09.904+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:10.272 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:09.904+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:10.272 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:09.904+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:10.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:10.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:10.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:10.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:10.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:10.478+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:10.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:10.479+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:11.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:10.883+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:11.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:10.883+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:11.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:10.883+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:11.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:10.884+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:11.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:10.925+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:11.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:10.925+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:11.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:10.925+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:11.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:10.926+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:11.807 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:11.501+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:11.807 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:11.501+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:11.808 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:11.501+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:11.808 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:11.502+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:12.087 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:03:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:03:11.965+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 46225 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:03:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:11.907+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:11.907+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:11.907+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:11.908+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:12.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:11.920+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:12.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:11.920+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:12.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:11.920+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:12.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:11.921+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:12.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:12.547+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:12.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:12.547+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:12.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:12.547+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:12.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:12.548+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:13.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:12.915+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:13.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:12.916+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:13.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:12.916+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:13.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:12.916+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:13.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:12.936+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:13.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:12.936+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:13.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:12.936+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:13.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:12.938+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:13.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:13.505+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:13.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:13.505+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:13.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:13.505+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:13.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:13.506+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:14.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:13.960+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:14.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:13.960+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:14.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:13.960+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:14.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:13.960+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:14.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:13.951+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:14.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:13.951+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:14.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:13.951+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:14.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:13.952+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:14.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:14.525+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:14.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:14.525+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:14.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:14.525+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:14.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:14.526+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:15.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:14.905+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:15.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:14.905+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:15.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:14.905+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:15.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:14.906+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:15.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:14.975+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:15.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:14.975+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:15.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:14.975+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:15.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:14.975+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:15.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:15.575+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:15.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:15.575+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:15.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:15.575+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:15.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:15.576+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:16.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:15.945+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:16.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:15.945+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:16.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:15.945+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:16.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:15.946+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:16.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:15.996+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:16.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:15.996+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:16.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:15.996+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:16.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:15.996+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:16.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:16.554+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:16.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:16.554+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:16.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:16.554+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:16.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:16.554+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:17.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:03:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:03:16.970+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 46272 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:03:17.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:16.954+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:17.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:16.954+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:17.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:16.954+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:17.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:16.955+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:17.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:17.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:17.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:17.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:17.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:17.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:17.032+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:17.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:17.519+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:17.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:17.519+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:17.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:17.519+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:17.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:17.520+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:18.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:17.978+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:18.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:17.978+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:18.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:17.978+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:18.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:17.979+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:18.062+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:18.062+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:18.062+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:18.062+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:18.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:18.493+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:18.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:18.493+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:18.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:18.493+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:18.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:18.493+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:19.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:19.109+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:19.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:19.109+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:19.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:19.109+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:19.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:19.110+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:19.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:18.999+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:19.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:18.999+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:19.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:18.999+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:19.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:19.000+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:19.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:19.496+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:19.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:19.496+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:19.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:19.496+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:19.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:19.496+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:20.271 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:19.987+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:20.271 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:19.987+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:20.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:19.987+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:20.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:19.988+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:20.272 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:20.078+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:20.272 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:20.078+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:20.272 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:20.078+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:20.273 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:20.079+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:20.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:20.533+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:20.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:20.533+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:20.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:20.533+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:20.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:20.534+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:21.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:21.064+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:21.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:21.064+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:21.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:21.064+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:21.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:21.065+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:21.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:21.005+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:21.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:21.006+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:21.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:21.006+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:21.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:21.007+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:21.807 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:21.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:21.808 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:21.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:21.808 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:21.506+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:21.808 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:21.507+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:22.087 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:03:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:03:21.976+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 46319 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:03:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:22.013+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:22.013+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:22.013+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:22.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:22.014+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:22.044+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:22.044+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:22.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:22.044+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:22.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:22.045+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:22.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:22.547+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:22.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:22.547+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:22.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:22.547+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:22.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:22.548+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:23.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:22.973+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:23.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:22.973+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:23.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:22.973+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:23.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:22.974+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:23.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:23.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:23.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:23.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:23.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:23.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:23.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:23.001+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:23.936 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:23.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:23.937 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:23.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:23.937 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:23.588+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:23.937 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:23.589+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:24.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:23.936+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:24.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:23.936+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:24.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:23.936+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:24.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:23.938+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:24.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:23.972+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:24.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:23.972+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:24.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:23.972+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:24.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:23.972+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:24.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:24.570+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:24.570+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:24.570+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:24.571+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:25.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:24.975+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:25.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:24.975+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:25.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:24.975+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:25.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:24.976+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:25.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:24.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:25.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:24.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:25.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:24.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:25.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:24.958+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:25.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:25.545+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:25.545+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:25.545+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:25.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:25.546+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:26.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:25.952+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:26.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:25.952+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:26.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:25.952+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:26.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:25.952+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:26.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:25.976+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:26.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:25.976+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:26.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:25.976+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:26.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:25.977+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:26.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:26.553+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:26.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:26.553+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:26.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:26.553+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:26.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:26.553+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:27.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:03:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:03:26.982+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 46365 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:03:27.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:27.001+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:27.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:27.001+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:27.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:27.001+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:27.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:27.002+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:26.934+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:26.934+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:26.934+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:27.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:26.934+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:27.913 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:27.603+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:27.913 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:27.603+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:27.913 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:27.603+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:27.913 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:27.603+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:28.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:27.987+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:28.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:27.987+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:28.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:27.987+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:28.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:27.988+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:28.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:27.913+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:28.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:27.913+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:28.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:27.913+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:28.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:27.914+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:28.884 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:28.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:28.884 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:28.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:28.885 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:28.609+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:28.885 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:28.609+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:29.244 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:29.037+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:29.244 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:29.037+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:29.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:29.037+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:29.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:29.037+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:29.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:28.884+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:29.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:28.884+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:29.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:28.884+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:29.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:28.885+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:29.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:29.577+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:29.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:29.577+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:29.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:29.577+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:29.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:29.578+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:30.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:30.031+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:30.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:30.031+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:30.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:30.031+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:30.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:30.032+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:30.273 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:29.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:30.273 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:29.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:30.273 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:29.852+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:30.273 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:29.853+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:30.818 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:30.565+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:30.819 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:30.565+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:30.819 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:30.565+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:30.819 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:30.566+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:31.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:31.066+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:31.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:31.066+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:31.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:31.066+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:31.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:31.067+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:31.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:30.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:31.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:30.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:31.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:30.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:31.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:30.819+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:31.808 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:31.541+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:31.808 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:31.541+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:31.808 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:31.541+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:31.809 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:31.542+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:32.078 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:03:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:03:31.988+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 46412 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:03:32.079 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:31.839+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:32.079 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:31.839+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:32.079 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:31.839+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:32.079 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:31.840+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:32.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:32.078+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:32.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:32.078+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:32.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:32.078+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:32.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:32.079+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:32.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:32.516+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:32.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:32.516+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:32.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:32.516+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:32.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:32.517+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:33.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:32.862+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:33.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:32.862+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:33.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:32.862+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:33.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:32.863+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:33.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:33.087+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:33.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:33.087+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:33.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:33.087+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:33.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:33.087+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:33.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:33.561+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:33.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:33.561+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:33.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:33.561+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:33.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:33.562+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:34.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:34.080+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:34.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:34.080+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:34.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:34.080+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:34.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:34.081+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:33.837+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:33.837+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:33.837+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:33.838+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:34.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:34.537+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:34.796 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:34.537+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:34.796 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:34.537+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:34.796 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:34.537+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:35.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:34.796+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:35.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:34.796+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:34.796+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:34.797+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:35.539 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:35.093+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:35.539 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:35.093+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:35.540 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:35.093+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:35.540 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:35.094+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:35.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:35.539+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:35.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:35.539+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:35.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:35.539+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:35.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:35.540+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:35.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:35.780+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:35.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:35.780+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:35.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:35.780+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:35.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:35.781+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:36.579 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:36.129+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:36.580 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:36.129+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:36.580 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:36.129+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:36.580 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:36.130+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:36.579+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:36.579+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:36.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:36.579+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:36.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:36.580+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:36.801+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:36.801+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:36.801+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:36.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:36.801+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:37.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:03:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:03:36.993+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 46457 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:03:37.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:37.113+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:37.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:37.113+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:37.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:37.113+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:37.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:37.113+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:38.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:37.625+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:38.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:37.625+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:38.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:37.625+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:38.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:37.626+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:38.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:37.789+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:38.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:37.789+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:38.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:37.789+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:38.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:37.790+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:38.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:38.110+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:38.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:38.110+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:38.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:38.110+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:38.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:38.111+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:39.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:38.783+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:39.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:38.783+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:39.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:38.783+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:39.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:38.784+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:39.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:38.637+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:39.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:38.637+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:39.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:38.637+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:39.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:38.638+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:39.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:39.095+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:39.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:39.095+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:39.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:39.095+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:39.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:39.096+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:40.079 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:39.593+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:40.079 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:39.593+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:40.079 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:39.593+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:40.079 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:39.594+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:40.079 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:39.833+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:40.080 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:39.833+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:40.080 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:39.833+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:40.080 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:39.834+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:40.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:40.079+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:40.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:40.079+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:40.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:40.079+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:40.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:40.080+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:40.859 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:40.597+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:40.859 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:40.597+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:40.859 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:40.597+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:40.859 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:40.598+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:41.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:41.077+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:41.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:41.077+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:41.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:41.077+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:41.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:41.078+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:41.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:40.859+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:41.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:40.859+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:41.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:40.859+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:41.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:40.860+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:42.000 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:41.630+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:42.000 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:41.630+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:42.000 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:41.630+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:42.000 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:41.631+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:42.001 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:41.838+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:42.001 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:41.838+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:42.001 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:41.838+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:42.001 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:41.838+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:42.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:03:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:03:42.000+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 46504 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:03:42.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:42.041+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:42.041+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:42.041+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:42.042+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:43.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:42.645+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:43.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:42.645+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:43.022 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:42.645+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:43.022 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:42.646+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:43.022 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:42.820+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:43.023 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:42.820+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:43.023 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:42.820+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:43.023 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:42.821+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:43.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:43.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:43.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:43.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:43.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:43.021+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:43.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:43.022+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:43.871 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:43.596+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:43.871 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:43.596+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:43.871 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:43.596+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:43.871 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:43.597+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:44.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:44.034+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:44.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:44.034+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:44.034+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:44.035+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:43.871+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:43.871+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:43.871+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:43.872+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:44.869 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:44.592+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:44.870 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:44.592+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:44.870 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:44.592+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:44.870 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:44.592+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:45.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:45.085+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:45.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:45.085+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:45.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:45.085+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:45.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:45.086+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:44.869+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:44.869+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:44.869+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:44.870+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:45.829 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:45.564+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:45.829 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:45.564+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:45.829 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:45.564+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:45.830 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:45.565+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:46.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:45.829+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:46.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:45.829+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:46.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:45.829+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:46.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:45.830+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:46.536 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:46.121+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:46.537 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:46.121+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:46.537 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:46.121+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:46.537 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:46.122+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:46.833 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:46.536+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:46.834 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:46.537+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:46.834 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:46.537+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:46.834 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:46.537+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:47.090 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:03:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:03:47.006+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 46550 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:03:47.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:46.834+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:47.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:46.834+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:47.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:46.834+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:47.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:46.835+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:47.584 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:47.100+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:47.584 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:47.100+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:47.584 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:47.100+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:47.585 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:47.101+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:47.837 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:47.826+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:47.837 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:47.827+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:47.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:47.827+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:47.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:47.827+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:47.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:47.584+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:47.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:47.584+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:47.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:47.584+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:47.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:47.585+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:48.541 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:48.143+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:48.542 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:48.143+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:48.542 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:48.143+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:48.542 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:48.144+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:48.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:48.541+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:48.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:48.541+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:48.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:48.541+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:48.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:48.542+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:48.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:48.780+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:48.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:48.780+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:48.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:48.780+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:48.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:48.781+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:49.517 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:49.120+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:49.518 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:49.120+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:49.518 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:49.120+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:49.518 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:49.121+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:49.815 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:49.518+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:49.815 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:49.518+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:49.815 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:49.518+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:49.816 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:49.519+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:50.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:49.815+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:49.815+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:49.815+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:49.816+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:50.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:50.126+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:50.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:50.126+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:50.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:50.126+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:50.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:50.127+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:50.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:50.474+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:50.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:50.474+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:50.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:50.474+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:50.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:50.475+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:51.134 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:50.847+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:51.134 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:50.847+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:51.134 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:50.847+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:51.134 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:50.848+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:51.447 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:51.134+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:51.447 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:51.134+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:51.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:51.134+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:51.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:51.134+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:51.810 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:51.447+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:51.810 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:51.447+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:51.810 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:51.447+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:51.810 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:51.448+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:52.088 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:03:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:03:52.012+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 46597 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:03:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:51.869+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:51.869+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:51.869+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:52.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:51.870+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:52.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:52.142+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:52.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:52.142+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:52.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:52.142+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:52.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:52.142+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:52.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:52.472+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:52.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:52.472+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:52.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:52.472+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:52.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:52.473+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:53.161 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:52.894+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:53.161 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:52.894+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:53.161 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:52.894+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:53.162 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:52.895+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:53.518 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:53.161+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:53.518 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:53.161+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:53.518 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:53.161+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:53.518 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:53.162+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:53.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:53.518+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:53.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:53.518+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:53.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:53.518+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:53.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:53.519+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:54.167 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:53.917+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:54.168 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:53.917+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:54.168 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:53.917+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:54.168 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:53.918+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:54.528 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:54.168+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:54.529 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:54.168+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:54.529 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:54.168+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:54.529 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:54.168+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:54.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:54.528+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:54.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:54.528+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:54.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:54.528+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:54.529+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:55.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:54.925+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:55.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:54.925+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:55.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:54.925+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:55.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:54.925+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:55.526 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:55.197+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:55.526 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:55.197+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:55.526 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:55.197+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:55.526 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:55.198+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:55.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:55.526+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:55.526+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:55.526+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:55.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:55.527+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:56.170 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:55.889+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:56.171 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:55.889+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:56.171 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:55.889+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:56.171 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:55.889+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:56.561 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:56.172+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:56.561 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:56.172+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:56.562 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:56.172+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:56.562 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:56.173+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:56.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:56.562+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:56.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:56.562+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:56.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:56.562+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:56.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:56.563+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:57.141 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:03:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:03:57.019+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 46642 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:03:57.141 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:56.847+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:57.141 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:56.847+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:57.142 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:56.847+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:57.142 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:56.848+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:57.565 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:57.146+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:57.566 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:57.146+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:57.566 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:57.146+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:57.566 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:57.147+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:57.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:57.566+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:57.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:57.566+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:57.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:57.566+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:57.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:57.567+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:58.141 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:57.871+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:58.142 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:57.871+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:58.142 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:57.871+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:58.142 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:57.872+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:58.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:58.143+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:58.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:58.143+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:58.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:58.143+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:58.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:58.144+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:58.912 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:58.593+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:58.912 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:58.593+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:58.913 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:58.593+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:58.913 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:58.594+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:59.244 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:59.133+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:59.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:59.133+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:59.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:59.133+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:03:59.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:03:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:03:59.134+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:03:59.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:58.913+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:59.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:58.913+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:59.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:58.913+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:03:59.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:58.914+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:03:59.963 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:59.620+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:59.963 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:59.620+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:59.963 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:59.620+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:03:59.964 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:03:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:03:59.621+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:00.274 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:00.160+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:00.274 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:00.160+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:00.275 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:00.160+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:00.275 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:00.160+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:00.275 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:59.964+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:00.275 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:59.964+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:00.275 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:59.964+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:00.275 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:03:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:03:59.964+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:00.978 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:00.637+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:00.978 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:00.637+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:00.979 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:00.637+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:00.979 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:00.638+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:01.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:01.167+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:01.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:01.167+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:01.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:01.167+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:01.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:01.168+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:00.979+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:00.979+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:00.979+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:00.980+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:02.017 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:01.626+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:02.017 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:01.626+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:02.017 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:01.626+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:02.017 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:01.627+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:02.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:04:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:04:02.025+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 46689 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:04:02.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:02.201+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:02.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:02.201+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:02.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:02.201+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:02.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:02.202+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:02.018+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:02.018+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:02.018+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:02.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:02.019+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:02.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:02.579+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:02.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:02.579+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:02.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:02.579+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:02.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:02.580+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:03.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:03.222+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:03.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:03.222+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:03.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:03.222+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:03.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:03.223+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:02.985+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:02.985+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:02.985+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:02.986+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:03.746 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:03.559+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:03.747 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:03.559+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:03.747 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:03.559+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:03.747 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:03.560+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:04.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:03.979+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:04.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:03.979+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:04.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:03.979+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:04.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:03.979+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:04.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:04.222+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:04.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:04.222+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:04.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:04.222+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:04.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:04.222+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:04.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:04.606+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:04.977 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:04.606+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:04.977 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:04.606+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:04.977 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:04.607+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:05.227 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:04.977+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:05.227 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:04.977+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:05.227 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:04.977+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:05.227 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:04.977+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:05.578 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:05.227+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:05.579 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:05.227+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:05.579 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:05.227+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:05.579 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:05.228+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:05.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:05.579+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:05.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:05.579+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:05.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:05.579+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:05.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:05.580+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:06.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:06.183+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:06.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:06.183+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:06.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:06.183+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:06.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:06.184+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:06.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:05.949+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:06.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:05.949+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:06.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:05.949+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:06.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:05.949+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:06.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:06.546+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:06.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:06.546+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:06.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:06.546+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:06.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:06.547+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:07.216 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:04:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:04:07.030+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 46735 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:04:07.216 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:06.949+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:07.216 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:06.949+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:07.216 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:06.949+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:07.216 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:06.949+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:07.563 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:07.216+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:07.563 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:07.216+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:07.563 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:07.216+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:07.563 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:07.217+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:07.564 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:07.563+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:07.564 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:07.563+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:07.564 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:07.563+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:07.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:07.564+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:08.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:07.941+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:08.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:07.942+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:08.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:07.942+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:07.943+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:08.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:08.180+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:08.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:08.180+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:08.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:08.180+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:08.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:08.181+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:08.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:08.578+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:08.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:08.578+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:08.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:08.578+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:08.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:08.579+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:09.244 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:09.216+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:09.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:09.216+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:09.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:09.216+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:09.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:09.216+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:09.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:08.983+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:09.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:08.983+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:09.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:08.983+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:09.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:08.984+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:09.941 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:09.616+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:09.941 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:09.616+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:09.941 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:09.616+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:09.941 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:09.616+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:10.265 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:09.941+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:10.266 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:09.941+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:10.266 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:09.941+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:10.266 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:09.942+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:10.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:10.265+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:10.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:10.265+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:10.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:10.265+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:10.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:10.266+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:10.929 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:10.656+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:10.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:10.656+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:10.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:10.656+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:10.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:10.657+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:11.310 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:10.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:11.311 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:10.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:11.311 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:10.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:11.311 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:10.930+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:11.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:11.310+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:11.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:11.310+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:11.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:11.310+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:11.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:11.311+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:11.945 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:11.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:11.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:11.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:11.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:11.642+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:11.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:11.643+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:12.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:04:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:04:12.036+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 46781 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:04:12.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:11.946+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:12.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:11.946+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:12.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:11.946+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:12.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:11.946+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:12.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:12.356+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:12.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:12.356+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:12.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:12.356+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:12.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:12.356+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:12.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:12.633+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:12.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:12.633+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:12.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:12.633+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:12.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:12.634+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:13.335 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:12.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:13.335 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:12.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:13.336 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:12.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:13.336 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:12.931+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:13.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:13.335+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:13.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:13.335+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:13.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:13.335+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:13.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:13.336+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:13.929 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:13.645+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:13.929 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:13.645+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:13.929 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:13.645+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:13.929 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:13.646+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:14.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:13.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:14.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:13.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:14.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:13.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:14.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:13.930+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:14.627 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:14.367+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:14.627 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:14.367+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:14.627 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:14.367+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:14.627 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:14.368+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:14.970 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:14.627+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:14.970 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:14.627+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:14.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:14.627+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:14.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:14.628+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:15.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:14.970+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:15.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:14.970+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:15.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:14.970+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:15.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:14.971+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:15.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:15.358+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:15.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:15.358+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:15.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:15.358+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:15.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:15.359+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:15.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:15.604+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:15.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:15.604+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:15.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:15.604+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:15.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:15.605+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:16.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:15.984+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:16.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:15.984+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:16.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:15.984+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:16.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:15.984+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:16.653 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:16.341+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:16.653 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:16.341+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:16.653 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:16.341+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:16.653 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:16.342+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:17.029 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:16.653+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:17.029 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:16.653+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:17.029 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:16.653+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:17.030 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:16.654+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:17.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:04:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:04:17.042+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 46827 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:04:17.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:17.029+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:17.029+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:17.029+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:17.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:17.030+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:17.689 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:17.341+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:17.689 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:17.341+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:17.689 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:17.341+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:17.689 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:17.342+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:18.051 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:17.689+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:18.051 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:17.689+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:18.051 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:17.689+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:18.051 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:17.690+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:18.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:18.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:18.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:18.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:18.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:18.051+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:18.052+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:18.734 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:18.353+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:18.734 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:18.353+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:18.734 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:18.353+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:18.734 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:18.354+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:19.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:18.734+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:19.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:18.734+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:19.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:18.734+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:19.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:18.735+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:19.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:19.319+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:19.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:19.319+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:19.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:19.319+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:19.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:19.320+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:19.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:19.089+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:19.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:19.089+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:19.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:19.089+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:19.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:19.090+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:20.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:19.762+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:20.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:19.762+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:20.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:19.762+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:20.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:19.762+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:20.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:20.279+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:20.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:20.279+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:20.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:20.279+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:20.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:20.280+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:20.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:20.129+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:20.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:20.129+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:20.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:20.129+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:20.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:20.130+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:21.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:20.792+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:21.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:20.792+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:21.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:20.792+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:21.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:20.793+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:21.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:21.274+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:21.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:21.274+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:21.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:21.274+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:21.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:21.275+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:21.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:21.088+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:21.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:21.088+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:21.088+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:21.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:21.089+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:22.048 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:21.781+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:22.048 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:21.781+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:22.048 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:21.781+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:22.049 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:21.782+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:22.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:04:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:04:22.048+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 46873 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:04:22.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:22.232+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:22.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:22.232+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:22.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:22.232+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:22.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:22.233+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:22.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:22.066+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:22.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:22.067+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:22.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:22.067+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:22.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:22.067+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:23.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:22.802+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:23.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:22.802+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:23.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:22.802+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:23.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:22.802+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:23.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:23.240+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:23.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:23.240+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:23.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:23.240+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:23.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:23.241+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:23.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:23.100+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:23.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:23.100+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:23.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:23.100+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:23.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:23.101+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:24.230 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:23.838+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:24.230 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:23.838+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:24.230 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:23.838+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:24.230 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:23.839+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:24.231 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:24.070+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:24.231 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:24.070+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:24.231 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:24.070+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:24.231 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:24.071+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:24.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:24.230+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:24.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:24.230+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:24.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:24.230+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:24.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:24.231+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:25.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:25.030+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:25.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:25.030+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:25.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:25.030+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:25.031+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:25.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:24.831+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:25.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:24.831+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:25.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:24.831+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:25.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:24.832+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:25.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:25.244+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:25.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:25.244+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:25.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:25.244+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:25.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:25.245+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:26.049 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:25.788+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:26.050 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:25.788+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:26.050 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:25.788+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:26.050 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:25.789+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:26.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:26.293+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:26.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:26.293+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:26.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:26.293+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:26.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:26.294+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:26.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:26.050+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:26.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:26.050+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:26.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:26.050+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:26.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:26.050+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:27.073 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:04:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:04:27.054+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 46920 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:04:27.073 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:26.822+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:27.073 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:26.822+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:27.074 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:26.822+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:27.074 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:26.823+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:27.074 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:27.025+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:27.074 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:27.025+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:27.074 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:27.025+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:27.074 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:27.026+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:27.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:27.282+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:27.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:27.282+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:27.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:27.282+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:27.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:27.283+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:28.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:27.809+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:28.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:27.809+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:28.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:27.809+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:28.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:27.810+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:28.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:27.978+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:28.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:27.978+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:28.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:27.978+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:28.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:27.979+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:28.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:28.263+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:28.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:28.263+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:28.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:28.263+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:28.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:28.265+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:29.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:28.785+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:29.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:28.785+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:29.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:28.785+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:29.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:28.786+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:28.938+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:28.938+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:28.938+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:28.939+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:29.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:29.313+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:29.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:29.313+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:29.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:29.313+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:29.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:29.314+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:30.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:29.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:30.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:29.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:29.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:29.907+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:30.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:29.827+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:30.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:29.827+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:30.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:29.827+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:30.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:29.828+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:30.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:30.274+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:30.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:30.274+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:30.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:30.274+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:30.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:30.275+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:31.228 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:30.850+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:31.229 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:30.850+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:31.229 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:30.850+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:31.229 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:30.851+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:31.229 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:30.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:31.229 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:30.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:31.230 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:30.906+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:31.230 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:30.906+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:31.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:31.229+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:31.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:31.229+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:31.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:31.229+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:31.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:31.230+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:32.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:04:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:04:32.060+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 46966 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:04:32.069 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:31.816+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:32.069 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:31.816+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:32.069 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:31.816+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:32.069 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:31.817+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:32.069 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:31.886+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:32.069 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:31.886+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:32.070 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:31.886+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:32.070 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:31.887+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:32.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:32.250+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:32.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:32.250+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:32.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:32.250+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:32.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:32.251+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:33.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:32.857+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:33.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:32.857+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:33.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:32.857+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:33.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:32.858+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:33.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:32.886+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:33.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:32.886+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:33.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:32.886+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:33.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:32.887+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:33.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:33.222+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:33.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:33.222+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:33.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:33.222+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:33.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:33.223+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:34.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:33.837+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:34.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:33.837+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:34.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:33.837+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:34.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:33.838+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:33.912+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:33.912+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:33.912+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:34.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:33.912+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:34.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:34.255+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:34.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:34.255+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:34.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:34.255+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:34.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:34.255+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:35.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:34.864+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:35.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:34.864+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:34.864+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:34.865+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:35.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:34.797+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:35.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:34.797+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:35.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:34.797+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:35.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:34.798+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:35.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:35.282+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:35.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:35.282+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:35.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:35.282+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:35.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:35.282+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:36.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:35.907+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:36.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:35.907+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:36.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:35.907+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:36.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:35.908+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:36.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:35.798+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:36.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:35.798+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:36.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:35.798+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:36.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:35.799+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:36.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:36.305+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:36.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:36.305+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:36.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:36.305+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:36.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:36.307+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:37.066 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:36.783+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:37.067 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:36.783+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:37.067 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:36.783+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:37.067 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:36.784+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:37.067 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:36.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:37.067 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:36.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:37.068 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:36.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:37.068 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:36.930+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:37.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:04:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:04:37.066+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 47012 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:04:37.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:37.298+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:37.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:37.298+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:37.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:37.298+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:37.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:37.299+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:38.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:37.822+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:38.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:37.822+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:38.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:37.822+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:38.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:37.824+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:38.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:37.890+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:38.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:37.890+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:38.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:37.890+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:38.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:37.891+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:38.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:38.338+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:38.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:38.338+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:38.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:38.338+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:38.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:38.339+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:39.244 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:38.850+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:39.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:38.850+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:39.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:38.850+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:39.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:38.851+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:39.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:38.888+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:39.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:38.888+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:39.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:38.888+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:39.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:38.888+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:39.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:39.388+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:39.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:39.388+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:39.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:39.388+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:39.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:39.389+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:40.277 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:39.839+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:40.278 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:39.839+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:40.278 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:39.839+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:40.278 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:39.840+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:40.278 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:39.922+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:40.279 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:39.922+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:40.279 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:39.922+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:40.279 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:39.922+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:40.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:40.351+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:40.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:40.351+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:40.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:40.351+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:40.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:40.351+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:41.330 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:40.863+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:41.330 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:40.863+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:41.330 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:40.863+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:41.330 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:40.864+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:41.331 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:40.962+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:41.331 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:40.962+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:41.331 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:40.962+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:41.331 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:40.963+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:41.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:41.330+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:41.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:41.330+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:41.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:41.330+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:41.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:41.330+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:42.072 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:41.848+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:42.072 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:41.848+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:42.072 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:41.848+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:42.072 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:41.849+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:42.073 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:42.004+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:42.073 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:42.004+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:42.073 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:42.004+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:42.073 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:42.005+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:42.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:04:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:04:42.072+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 47058 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:04:42.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:42.368+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:42.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:42.368+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:42.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:42.368+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:42.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:42.369+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:43.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:43.009+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:43.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:43.009+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:43.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:43.009+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:43.010+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:43.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:42.870+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:43.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:42.870+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:43.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:42.870+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:43.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:42.871+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:43.683 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:43.359+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:43.683 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:43.359+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:43.683 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:43.359+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:43.683 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:43.360+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:44.059 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:43.829+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:44.059 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:43.829+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:44.059 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:43.829+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:44.059 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:43.830+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:44.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:44.059+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:44.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:44.059+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:44.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:44.059+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:44.060+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:44.802 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:44.381+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:44.803 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:44.381+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:44.803 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:44.381+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:44.803 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:44.382+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:45.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:44.803+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:45.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:44.803+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:44.803+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:44.803+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:45.382 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:45.101+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:45.382 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:45.101+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:45.382 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:45.101+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:45.382 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:45.102+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:45.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:45.382+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:45.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:45.382+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:45.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:45.382+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:45.801 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:45.383+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:46.054 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:45.801+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:46.054 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:45.801+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:46.054 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:45.801+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:46.054 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:45.802+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:46.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:46.054+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:46.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:46.054+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:46.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:46.054+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:46.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:46.055+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:46.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:46.356+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:46.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:46.356+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:46.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:46.356+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:46.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:46.357+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:47.099 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:04:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:04:47.078+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 47105 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:04:47.100 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:46.840+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:47.100 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:46.840+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:47.100 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:46.840+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:47.100 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:46.841+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:47.100 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:47.022+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:47.100 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:47.022+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:47.101 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:47.022+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:47.101 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:47.023+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:47.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:47.313+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:47.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:47.313+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:47.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:47.313+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:47.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:47.314+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:48.300 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:47.883+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:48.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:47.883+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:48.335 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:47.883+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:48.335 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:47.884+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:48.335 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:48.004+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:48.336 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:48.004+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:48.336 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:48.004+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:48.336 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:48.005+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:48.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:48.300+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:48.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:48.300+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:48.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:48.300+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:48.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:48.301+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:49.244 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:48.961+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:49.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:48.961+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:49.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:48.961+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:49.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:48.962+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:49.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:48.884+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:49.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:48.884+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:49.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:48.884+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:49.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:48.885+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:49.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:49.323+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:49.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:49.323+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:49.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:49.323+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:49.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:49.324+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:50.278 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:49.850+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:50.278 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:49.850+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:50.278 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:49.850+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:50.278 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:49.851+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:50.279 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:49.991+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:50.279 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:49.991+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:50.279 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:49.991+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:50.279 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:49.992+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:50.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:50.342+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:50.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:50.342+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:50.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:50.342+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:50.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:50.343+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:51.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:51.015+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:51.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:51.015+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:51.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:51.015+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:51.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:51.016+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:50.832+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:50.832+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:50.832+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:50.832+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:51.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:51.297+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:51.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:51.297+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:51.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:51.297+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:51.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:51.298+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:52.084 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:51.810+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:52.084 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:51.810+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:52.084 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:51.810+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:52.084 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:51.810+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:52.084 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:52.010+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:52.085 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:52.010+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:52.085 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:52.010+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:52.085 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:52.010+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:52.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:04:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:04:52.084+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 47151 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:04:52.833 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:52.348+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:52.834 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:52.348+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:52.834 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:52.348+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:52.834 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:52.349+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:53.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:52.833+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:53.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:52.833+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:53.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:52.833+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:53.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:52.834+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:52.992+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:52.992+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:52.992+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:52.993+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:53.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:53.304+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:53.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:53.304+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:53.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:53.304+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:53.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:53.305+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:54.037 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:53.787+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:54.038 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:53.787+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:54.038 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:53.787+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:54.038 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:53.788+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:54.326 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:54.037+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:54.326 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:54.037+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:54.326 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:54.037+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:54.327 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:54.038+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:54.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:54.326+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:54.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:54.326+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:54.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:54.326+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:54.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:54.327+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:55.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:54.817+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:55.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:54.817+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:55.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:54.817+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:55.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:54.817+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:55.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:54.992+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:55.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:54.992+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:55.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:54.992+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:55.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:54.993+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:55.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:55.337+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:55.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:55.337+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:55.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:55.337+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:55.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:55.338+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:56.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:55.776+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:56.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:55.776+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:56.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:55.776+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:56.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:55.777+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:56.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:55.963+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:56.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:55.963+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:56.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:55.963+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:56.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:55.964+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:56.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:56.302+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:56.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:56.302+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:56.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:56.302+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:56.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:56.303+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:57.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:56.777+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:57.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:56.777+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:57.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:56.777+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:57.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:56.778+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:56.971+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:56.971+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:56.971+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:57.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:56.971+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:57.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:04:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:04:57.090+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 47197 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:04:57.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:57.295+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:57.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:57.295+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:57.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:57.295+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:57.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:57.296+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:58.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:57.805+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:58.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:57.805+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:58.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:57.805+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:58.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:57.806+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:58.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:57.932+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:58.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:57.932+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:58.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:57.932+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:58.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:57.933+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:58.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:58.259+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:58.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:58.259+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:58.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:58.259+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:58.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:58.260+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:04:59.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:58.827+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:59.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:58.827+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:59.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:58.827+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:04:59.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:58.828+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:58.912+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:58.912+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:58.912+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:04:59.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:58.912+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:04:59.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:59.300+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:59.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:59.300+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:59.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:59.300+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:04:59.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:04:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:04:59.301+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:00.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:59.797+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:00.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:59.797+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:00.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:59.797+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:00.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:04:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:04:59.798+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:00.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:59.943+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:00.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:59.943+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:00.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:59.943+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:00.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:04:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:04:59.943+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:00.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:00.259+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:00.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:00.259+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:00.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:00.259+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:00.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:00.260+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:01.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:00.755+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:01.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:00.755+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:01.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:00.755+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:01.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:00.756+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:01.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:00.950+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:01.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:00.950+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:01.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:00.950+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:01.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:00.951+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:01.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:01.289+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:01.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:01.289+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:01.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:01.289+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:01.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:01.290+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:02.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:01.722+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:02.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:01.722+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:02.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:01.722+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:02.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:01.723+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:01.919+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:01.919+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:01.919+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:01.919+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:02.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:05:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:05:02.096+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 47243 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:05:02.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:02.305+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:02.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:02.305+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:02.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:02.305+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:02.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:02.306+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:02.953 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:02.696+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:02.953 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:02.696+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:02.953 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:02.696+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:02.953 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:02.697+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:03.314 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:02.953+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:03.315 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:02.953+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:03.315 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:02.953+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:03.315 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:02.954+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:03.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:03.315+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:03.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:03.315+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:03.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:03.315+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:03.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:03.316+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:04.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:03.696+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:04.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:03.696+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:04.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:03.696+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:04.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:03.697+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:04.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:03.935+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:04.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:03.935+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:04.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:03.935+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:04.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:03.936+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:04.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:04.359+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:04.726 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:04.359+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:04.726 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:04.359+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:04.726 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:04.360+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:05.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:04.972+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:05.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:04.972+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:05.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:04.972+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:05.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:04.973+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:05.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:04.725+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:05.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:04.725+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:05.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:04.725+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:05.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:04.726+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:05.757 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:05.350+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:05.757 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:05.350+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:05.757 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:05.350+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:05.757 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:05.351+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:06.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:05.757+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:06.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:05.757+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:06.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:05.757+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:06.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:05.758+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:06.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:05.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:06.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:05.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:06.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:05.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:06.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:05.931+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:06.714 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:06.360+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:06.714 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:06.360+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:06.714 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:06.360+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:06.714 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:06.360+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:07.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:06.895+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:07.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:06.895+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:07.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:06.895+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:07.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:06.896+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:07.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:06.714+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:07.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:06.714+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:07.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:06.714+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:07.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:06.715+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:07.377 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:05:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:05:07.102+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 47290 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:05:07.756 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:07.377+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:07.756 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:07.377+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:07.757 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:07.377+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:07.757 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:07.378+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:08.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:07.756+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:08.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:07.756+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:08.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:07.756+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:08.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:07.757+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:08.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:07.934+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:08.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:07.934+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:08.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:07.934+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:08.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:07.934+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:08.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:08.404+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:08.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:08.404+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:08.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:08.404+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:08.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:08.405+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:09.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:08.723+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:09.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:08.723+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:09.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:08.723+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:09.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:08.725+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:09.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:08.902+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:09.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:08.902+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:09.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:08.902+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:09.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:08.902+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:09.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:09.380+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:09.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:09.380+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:09.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:09.380+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:09.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:09.381+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:10.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:09.734+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:10.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:09.734+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:10.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:09.734+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:10.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:09.735+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:10.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:09.904+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:10.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:09.904+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:10.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:09.904+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:10.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:09.905+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:10.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:10.348+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:10.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:10.348+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:10.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:10.348+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:10.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:10.349+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:11.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:10.745+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:11.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:10.745+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:11.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:10.745+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:11.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:10.746+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:11.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:10.919+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:11.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:10.919+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:10.919+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:11.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:10.920+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:11.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:11.378+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:11.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:11.378+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:11.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:11.378+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:11.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:11.379+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:12.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:11.951+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:11.951+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:11.951+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:11.952+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:12.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:11.769+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:12.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:11.769+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:12.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:11.769+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:12.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:11.769+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:12.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:05:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:05:12.107+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 47336 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:05:12.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:12.330+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:12.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:12.330+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:12.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:12.330+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:12.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:12.332+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:13.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:12.789+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:13.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:12.789+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:13.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:12.789+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:13.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:12.790+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:12.912+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:12.912+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:12.912+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:12.913+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:13.764 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:13.344+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:13.764 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:13.344+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:13.764 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:13.344+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:13.764 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:13.345+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:14.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:13.764+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:14.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:13.764+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:14.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:13.764+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:14.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:13.765+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:14.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:13.882+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:14.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:13.882+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:14.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:13.882+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:14.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:13.883+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:14.797 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:14.385+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:14.797 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:14.385+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:14.798 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:14.385+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:14.798 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:14.385+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:15.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:14.798+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:15.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:14.798+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:15.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:14.798+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:15.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:14.798+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:15.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:14.922+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:15.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:14.922+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:15.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:14.922+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:15.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:14.922+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:15.780 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:15.420+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:15.780 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:15.420+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:15.780 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:15.420+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:15.781 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:15.421+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:16.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:15.779+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:16.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:15.779+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:16.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:15.779+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:16.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:15.781+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:16.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:15.884+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:16.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:15.884+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:16.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:15.884+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:16.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:15.884+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:16.760 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:16.419+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:16.761 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:16.419+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:16.761 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:16.419+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:16.761 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:16.420+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:17.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:16.761+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:17.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:16.762+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:17.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:16.762+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:17.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:16.763+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:16.855+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:16.855+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:16.855+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:16.856+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:17.408 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:05:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:05:17.114+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 47382 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:05:17.790 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:17.410+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:17.791 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:17.410+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:17.791 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:17.410+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:17.791 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:17.411+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:18.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:17.792+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:18.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:17.792+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:18.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:17.792+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:18.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:17.793+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:18.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:17.830+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:18.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:17.830+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:18.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:17.830+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:18.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:17.831+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:18.794 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:18.436+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:18.794 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:18.436+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:18.794 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:18.436+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:18.794 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:18.438+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:19.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:18.845+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:19.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:18.845+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:19.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:18.845+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:19.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:18.846+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:19.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:18.795+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:19.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:18.795+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:19.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:18.795+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:19.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:18.796+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:19.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:19.471+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:19.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:19.471+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:19.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:19.471+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:19.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:19.472+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:20.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:19.824+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:20.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:19.824+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:20.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:19.824+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:20.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:19.825+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:20.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:19.780+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:20.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:19.780+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:20.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:19.780+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:20.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:19.781+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:20.546 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:20.444+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:20.547 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:20.444+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:20.547 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:20.444+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:20.547 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:20.445+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:20.796 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:20.798+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:20.797 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:20.798+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:21.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:20.798+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:21.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:20.798+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:21.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:20.824+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:21.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:20.824+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:21.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:20.824+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:21.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:20.825+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:21.767 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:21.410+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:21.767 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:21.410+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:21.768 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:21.410+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:21.768 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:21.411+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:22.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:21.792+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:22.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:21.792+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:22.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:21.792+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:22.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:21.792+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:21.768+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:21.768+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:21.768+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:21.769+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:22.374 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:05:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:05:22.120+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 47428 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:05:22.776 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:22.374+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:22.776 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:22.374+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:22.776 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:22.374+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:22.777 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:22.375+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:23.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:22.833+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:23.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:22.833+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:23.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:22.833+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:23.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:22.834+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:22.777+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:22.777+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:22.777+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:23.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:22.778+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:23.166 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-03-19T04:05:23.172 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T04:05:23.172 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-03-19T04:05:23.173 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-03-19T04:05:23.187 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-19T04:05:23.198 INFO:tasks.cephadm:Teardown begin 2024-03-19T04:05:23.198 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephadm.py", line 1867, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-03-19T04:05:23.203 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T04:05:23.237 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:05:23.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:05:23.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:23.343+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:23.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:23.343+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:23.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:23.343+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:23.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:23.344+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:24.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:23.837+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:24.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:23.837+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:24.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:23.837+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:24.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:23.838+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:24.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:23.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:24.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:23.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:23.794+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:24.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:23.795+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:24.818 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:24.343+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:24.818 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:24.343+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:24.818 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:24.343+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:24.818 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:24.344+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:25.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:24.861+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:25.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:24.861+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:25.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:24.861+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:25.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:24.862+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:24.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:24.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:24.818+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:24.819+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:25.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:25.321+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:25.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:25.321+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:25.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:25.321+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:25.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:25.322+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:26.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:25.771+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:26.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:25.771+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:26.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:25.771+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:26.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:25.772+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:26.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:25.909+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:26.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:25.909+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:26.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:25.909+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:26.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:25.909+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:26.304 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:05:26.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:26.300+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:26.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:26.300+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:26.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:26.300+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:26.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:26.301+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:27.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:26.738+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:27.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:26.738+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:27.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:26.738+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:27.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:26.738+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:27.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:26.871+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:27.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:26.871+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:27.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:26.871+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:27.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:26.872+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:27.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:05:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:05:27.125+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 47475 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:05:27.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:27.308+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:27.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:27.308+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:27.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:27.308+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:27.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:27.309+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:28.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:27.825+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:28.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:27.825+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:28.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:27.825+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:28.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:27.826+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:28.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:27.723+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:28.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:27.723+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:28.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:27.723+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:28.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:27.724+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:28.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:28.288+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:28.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:28.288+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:28.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:28.288+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:28.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:28.289+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:29.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:28.835+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:29.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:28.835+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:29.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:28.835+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:29.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:28.835+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:28.720+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:28.720+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:28.720+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:29.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:28.721+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:29.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:29.314+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:29.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:29.314+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:29.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:29.314+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:29.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:29.315+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:30.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:29.796+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:30.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:29.796+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:30.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:29.796+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:30.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:29.797+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:29.757+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:29.757+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:29.757+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:29.758+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:30.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:30.300+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:30.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:30.300+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:30.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:30.300+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:30.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:30.301+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:31.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:30.829+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:31.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:30.829+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:31.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:30.829+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:31.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:30.830+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:31.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:30.799+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:31.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:30.799+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:31.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:30.799+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:31.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:30.800+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:31.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:31.295+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:31.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:31.295+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:31.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:31.295+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:31.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:31.296+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:32.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:31.860+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:32.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:31.860+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:32.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:31.860+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:32.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:31.861+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:31.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:31.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:31.763+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:32.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:31.763+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:32.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:05:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:05:32.132+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 47521 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:05:32.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:32.330+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:32.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:32.330+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:32.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:32.330+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:32.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:32.331+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:33.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:32.899+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:33.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:32.899+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:33.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:32.899+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:33.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:32.900+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:32.802+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:32.802+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:33.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:32.802+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:33.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:32.803+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:33.398 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:33.297+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:33.399 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:33.297+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:33.399 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:33.297+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:33.399 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:33.297+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:33.758 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:33.758+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:33.758 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:33.758+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:33.758 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:33.758+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:34.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:33.759+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:34.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:33.905+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:34.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:33.905+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:34.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:33.905+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:34.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:33.906+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:34.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:34.249+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:34.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:34.249+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:34.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:34.249+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:34.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:34.250+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:35.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:34.894+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:35.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:34.894+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:35.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:34.894+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:35.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:34.894+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:34.754+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:34.754+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:34.754+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:35.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:34.754+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:35.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:05:35.305 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:05:35.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:35.273+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:35.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:35.273+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:35.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:35.273+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:35.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:35.274+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:36.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:35.927+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:36.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:35.927+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:36.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:35.927+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:36.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:35.928+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:36.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:35.783+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:36.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:35.783+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:36.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:35.783+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:36.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:35.784+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:36.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:36.256+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:36.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:36.256+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:36.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:36.256+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:36.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:36.257+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:37.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:36.908+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:37.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:36.908+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:37.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:36.908+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:37.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:36.909+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:36.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:36.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:36.755+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:37.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:36.756+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:37.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:05:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:05:37.138+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 47567 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:05:37.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:37.260+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:37.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:37.260+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:37.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:37.260+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:37.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:37.261+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:38.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:37.929+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:38.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:37.929+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:38.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:37.929+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:38.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:37.930+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:38.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:37.754+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:38.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:37.754+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:38.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:37.754+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:38.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:37.754+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:38.366 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:05:38.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:38.265+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:38.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:38.265+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:38.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:38.265+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:38.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:38.265+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:39.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:38.940+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:39.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:38.940+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:39.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:38.940+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:39.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:38.941+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:38.771+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:39.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:38.771+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:39.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:38.771+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:39.090 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:38.771+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:39.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:39.314+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:39.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:39.314+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:39.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:39.314+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:39.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:39.315+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:40.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:39.786+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:40.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:39.786+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:40.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:39.786+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:40.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:39.787+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:40.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:39.891+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:40.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:39.891+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:40.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:39.891+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:40.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:39.892+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:40.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:40.268+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:40.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:40.268+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:40.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:40.268+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:40.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:40.269+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:41.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:40.936+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:41.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:40.936+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:41.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:40.936+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:41.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:40.937+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:41.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:40.787+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:41.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:40.787+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:41.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:40.787+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:41.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:40.788+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:41.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:41.258+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:41.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:41.258+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:41.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:41.258+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:41.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:41.259+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:42.051 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:41.984+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:42.051 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:41.984+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:42.051 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:41.984+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:42.051 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:41.985+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:42.051 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:41.745+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:42.052 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:41.745+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:42.052 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:41.745+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:42.052 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:41.745+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:42.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:05:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:05:42.145+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 47613 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:05:42.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:42.291+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:42.291+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:42.291+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:42.291+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:43.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:42.949+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:43.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:42.949+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:43.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:42.950+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:43.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:42.951+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:43.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:42.703+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:43.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:42.703+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:43.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:42.703+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:43.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:42.703+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:43.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:43.277+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:43.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:43.277+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:43.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:43.277+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:43.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:43.278+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:44.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:43.749+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:44.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:43.749+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:44.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:43.749+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:44.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:43.750+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:44.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:43.961+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:44.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:43.961+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:44.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:43.961+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:44.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:43.961+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:44.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:44.308+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:44.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:44.308+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:44.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:44.308+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:44.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:44.309+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:45.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:44.943+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:45.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:44.943+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:44.943+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:44.944+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:45.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:44.706+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:45.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:44.706+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:45.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:44.706+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:45.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:44.707+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:45.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:45.316+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:45.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:45.316+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:45.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:45.316+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:45.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:45.317+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:46.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:45.896+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:46.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:45.896+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:46.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:45.896+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:46.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:45.896+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:46.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:45.736+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:46.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:45.736+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:46.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:45.736+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:46.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:45.737+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:46.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:46.342+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:46.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:46.342+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:46.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:46.342+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:46.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:46.343+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:47.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:46.864+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:47.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:46.864+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:47.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:46.864+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:47.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:46.865+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:47.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:46.723+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:47.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:46.723+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:47.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:46.723+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:47.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:46.724+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:47.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:05:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:05:47.150+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 47660 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:05:47.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:47.364+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:47.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:47.364+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:47.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:47.364+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:47.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:47.366+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:48.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:47.878+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:48.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:47.878+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:48.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:47.878+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:48.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:47.879+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:48.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:47.716+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:48.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:47.716+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:48.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:47.716+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:48.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:47.717+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:48.739 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:48.394+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:48.739 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:48.394+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:48.739 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:48.394+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:48.739 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:48.395+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:49.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:48.871+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:49.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:48.871+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:49.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:48.871+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:49.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:48.872+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:49.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:48.739+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:49.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:48.739+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:49.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:48.739+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:49.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:48.740+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:49.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:49.398+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:49.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:49.398+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:49.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:49.398+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:49.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:49.399+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:50.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:49.917+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:50.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:49.917+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:50.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:49.917+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:50.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:49.918+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:49.714+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:49.714+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:49.714+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:50.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:49.715+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:50.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:05:50.369 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:05:50.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:50.376+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:50.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:50.376+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:50.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:50.376+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:50.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:50.377+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:51.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:50.699+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:51.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:50.699+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:51.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:50.699+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:51.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:50.700+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:50.928+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:50.928+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:50.928+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:51.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:50.929+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:51.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:51.403+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:51.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:51.403+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:51.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:51.403+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:51.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:51.403+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:52.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:51.740+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:51.740+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:51.740+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:51.741+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:52.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:51.926+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:52.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:51.926+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:51.926+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:52.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:51.927+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:52.452 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:05:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:05:52.156+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 47706 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:05:52.788 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:52.453+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:52.788 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:52.453+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:52.788 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:52.453+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:52.789 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:52.453+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:53.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:52.969+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:53.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:52.969+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:53.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:52.969+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:53.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:52.970+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:52.788+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:52.788+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:52.788+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:52.789+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:53.442 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:05:53.759 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:53.429+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:53.759 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:53.429+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:53.759 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:53.429+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:53.759 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:53.430+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:54.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:53.940+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:54.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:53.940+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:54.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:53.940+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:54.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:53.941+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:54.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:53.759+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:54.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:53.759+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:54.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:53.759+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:54.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:53.760+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:54.727 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:54.427+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:54.727 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:54.427+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:54.727 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:54.427+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:54.728 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:54.428+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:55.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:54.962+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:55.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:54.962+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:55.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:54.962+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:55.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:54.963+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:55.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:54.727+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:55.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:54.727+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:55.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:54.727+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:55.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:54.728+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:55.756 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:55.438+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:55.757 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:55.438+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:55.757 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:55.438+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:55.757 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:55.439+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:56.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:55.957+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:56.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:55.957+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:56.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:55.957+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:56.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:55.958+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:56.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:55.757+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:56.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:55.757+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:56.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:55.757+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:56.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:55.757+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:56.443 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:05:56.444 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:05:56.514 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:05:56.514 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-19T04:05:56.515 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-19T04:05:56.515 DEBUG:teuthology.orchestra.run.smithi059:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-19T04:05:56.532 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:05:56.532 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:05:56.806 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:56.486+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:56.806 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:56.486+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:56.806 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:56.486+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:56.806 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:56.487+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:57.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:56.912+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:57.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:56.912+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:57.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:56.912+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:57.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:56.913+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:56.806+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:56.806+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:56.806+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:56.807+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:57.520 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:05:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:05:57.162+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 47752 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:05:57.786 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:57.521+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:57.786 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:57.521+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:57.786 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:57.521+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:57.786 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:57.521+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:58.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:57.786+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:58.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:57.786+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:58.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:57.786+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:58.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:57.787+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:58.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:57.878+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:58.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:57.878+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:58.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:57.878+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:58.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:57.879+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:58.782 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:58.515+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:58.783 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:58.515+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:58.783 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:58.515+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:58.783 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:58.516+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:05:59.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:58.863+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:59.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:58.863+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:59.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:58.863+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:05:59.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:58.864+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:59.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:58.783+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:59.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:58.783+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:59.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:58.783+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:05:59.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:58.783+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:05:59.582 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:05:59.808 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:59.544+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:59.808 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:59.544+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:59.808 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:59.544+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:05:59.808 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:05:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:05:59.545+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:00.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:59.808+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:00.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:59.808+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:00.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:59.808+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:00.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:05:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:05:59.809+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:00.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:59.851+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:00.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:59.851+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:00.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:59.851+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:00.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:05:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:05:59.852+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:00.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:00.526+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:00.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:00.526+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:00.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:00.526+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:00.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:00.526+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:01.337 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:00.847+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:01.337 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:00.847+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:01.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:00.847+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:01.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:00.848+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:00.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:00.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:00.858+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:01.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:00.859+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:01.818 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:01.501+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:01.819 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:01.501+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:01.819 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:01.501+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:01.819 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:01.502+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:02.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:01.882+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:02.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:01.882+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:02.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:01.882+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:02.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:01.883+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:01.835+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:01.835+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:01.835+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:01.836+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:02.492 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:06:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:06:02.168+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 47798 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:06:02.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:02.493+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:02.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:02.493+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:02.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:02.493+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:02.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:02.494+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:03.337 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:02.854+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:03.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:02.854+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:03.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:02.854+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:03.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:02.855+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:03.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:02.875+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:02.875+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:02.875+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:03.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:02.875+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:03.830 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:03.474+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:03.830 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:03.474+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:03.831 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:03.474+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:03.831 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:03.476+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:04.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:03.830+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:04.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:03.830+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:04.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:03.830+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:04.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:03.832+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:04.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:03.889+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:04.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:03.889+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:04.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:03.889+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:04.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:03.890+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:04.791 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:04.462+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:04.792 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:04.462+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:04.792 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:04.462+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:04.792 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:04.463+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:05.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:04.791+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:05.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:04.791+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:05.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:04.791+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:05.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:04.793+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:05.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:04.928+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:05.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:04.928+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:05.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:04.928+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:05.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:04.929+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:05.747 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:05.461+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:05.748 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:05.461+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:05.748 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:05.461+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:05.748 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:05.462+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:06.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:05.748+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:06.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:05.748+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:06.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:05.748+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:06.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:05.749+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:06.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:05.948+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:06.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:05.948+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:06.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:05.948+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:06.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:05.948+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:06.778 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:06.481+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:06.778 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:06.481+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:06.778 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:06.481+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:06.778 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:06.482+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:07.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:06.778+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:07.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:06.778+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:07.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:06.778+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:07.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:06.779+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:07.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:06.983+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:07.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:06.983+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:07.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:06.983+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:07.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:06.984+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:07.492 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:06:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:06:07.174+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 47845 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:06:07.751 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:07.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:07.751 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:07.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:07.752 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:07.492+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:07.752 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:07.494+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:08.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:07.751+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:08.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:07.751+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:08.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:07.751+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:08.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:07.752+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:08.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:07.973+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:08.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:07.973+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:08.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:07.973+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:08.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:07.973+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:08.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:06:08.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:06:08.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:08.731+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:08.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:08.731+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:08.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:08.731+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:08.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:08.731+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:08.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:08.487+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:08.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:08.487+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:08.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:08.487+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:08.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:08.488+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:09.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:08.998+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:09.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:08.998+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:09.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:08.998+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:09.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:08.998+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:09.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:09.488+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:09.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:09.488+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:09.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:09.488+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:09.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:09.489+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:09.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:09.731+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:09.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:09.731+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:09.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:09.731+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:09.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:09.731+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:10.285 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:10.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:10.286 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:10.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:10.286 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:10.033+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:10.286 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:10.034+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:10.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:10.520+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:10.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:10.520+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:10.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:10.520+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:10.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:10.520+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:11.074 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:10.739+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:11.074 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:10.739+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:11.074 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:10.739+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:11.075 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:10.740+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:11.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:11.074+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:11.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:11.074+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:11.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:11.074+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:11.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:11.075+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:11.646 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:06:11.820 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:11.524+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:11.820 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:11.524+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:11.820 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:11.524+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:11.820 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:11.524+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:11.820 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:11.745+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:11.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:11.745+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:11.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:11.745+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:11.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:11.746+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:12.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:12.030+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:12.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:12.030+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:12.030+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:12.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:12.031+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:12.545 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:06:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:06:12.181+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 47891 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:06:12.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:12.721+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:12.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:12.721+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:12.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:12.721+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:12.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:12.721+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:12.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:12.545+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:12.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:12.545+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:12.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:12.545+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:12.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:12.546+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:13.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:13.017+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:13.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:13.017+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:13.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:13.017+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:13.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:13.018+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:13.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:13.562+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:13.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:13.562+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:13.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:13.562+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:13.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:13.563+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:13.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:13.713+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:13.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:13.713+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:13.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:13.713+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:13.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:13.714+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:14.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:13.998+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:14.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:13.998+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:14.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:13.998+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:14.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:13.998+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:14.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:14.576+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:14.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:14.576+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:14.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:14.576+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:14.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:14.577+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:14.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:14.681+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:14.841 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:14.681+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:14.841 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:14.681+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:14.842 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:14.682+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:15.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:15.013+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:15.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:15.013+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:15.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:15.013+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:15.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:15.014+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:15.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:15.700+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:15.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:15.700+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:15.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:15.700+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:15.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:15.701+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:15.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:15.546+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:15.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:15.546+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:15.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:15.546+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:15.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:15.547+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:16.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:15.970+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:16.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:15.970+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:16.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:15.970+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:16.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:15.971+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:16.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:16.676+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:16.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:16.676+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:16.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:16.676+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:16.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:16.676+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:16.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:16.539+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:16.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:16.539+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:16.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:16.539+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:16.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:16.540+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:17.187 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:16.922+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:17.188 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:16.922+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:17.188 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:16.922+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:17.188 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:16.922+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:17.532 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:06:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:06:17.187+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 47937 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:06:17.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:17.532+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:17.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:17.532+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:17.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:17.532+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:17.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:17.533+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:17.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:17.635+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:17.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:17.635+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:17.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:17.635+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:17.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:17.636+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:18.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:17.956+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:18.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:17.956+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:18.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:17.956+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:17.957+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:18.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:18.619+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:18.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:18.619+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:18.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:18.619+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:18.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:18.620+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:18.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:18.555+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:18.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:18.555+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:18.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:18.555+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:18.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:18.556+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:19.244 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:18.985+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:19.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:18.985+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:19.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:18.985+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:19.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:18.986+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:19.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:19.641+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:19.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:19.641+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:19.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:19.641+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:19.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:19.642+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:19.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:19.555+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:19.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:19.555+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:19.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:19.555+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:19.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:19.555+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:20.284 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:19.956+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:20.285 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:19.956+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:20.285 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:19.956+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:20.285 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:19.957+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:20.908 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:20.604+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:20.909 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:20.604+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:20.909 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:20.604+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:20.909 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:20.605+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:20.909 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:20.669+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:20.909 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:20.669+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:20.910 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:20.669+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:20.910 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:20.670+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:21.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:20.908+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:21.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:20.909+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:21.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:20.909+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:21.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:20.909+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:21.885 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:21.648+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:21.885 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:21.648+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:21.885 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:21.648+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:21.885 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:21.649+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:21.885 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:21.589+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:21.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:21.589+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:21.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:21.589+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:21.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:21.590+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:22.193 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:21.885+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:22.194 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:21.885+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:22.194 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:21.885+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:22.194 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:21.886+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:22.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:06:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:06:22.193+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 47983 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:06:23.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:22.624+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:23.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:22.624+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:22.624+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:22.625+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:23.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:22.638+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:23.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:22.638+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:23.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:22.638+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:23.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:22.639+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:23.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:22.872+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:23.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:22.872+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:23.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:22.872+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:23.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:22.873+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:23.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:06:23.648 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:06:23.917 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:23.602+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:23.918 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:23.602+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:23.918 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:23.602+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:23.918 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:23.603+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:23.918 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:23.646+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:23.919 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:23.646+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:23.919 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:23.646+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:23.919 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:23.647+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:24.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:23.917+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:24.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:23.917+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:24.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:23.917+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:24.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:23.918+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:24.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:24.554+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:24.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:24.554+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:24.554+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:24.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:24.554+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:24.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:24.686+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:24.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:24.686+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:24.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:24.686+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:24.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:24.687+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:25.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:24.885+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:25.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:24.885+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:25.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:24.885+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:25.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:24.886+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:25.523+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:25.523+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:25.523+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:25.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:25.524+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:25.671+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:25.671+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:25.671+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:25.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:25.672+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:26.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:25.916+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:26.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:25.916+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:26.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:25.916+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:26.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:25.917+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:26.718 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:06:26.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:26.640+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:26.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:26.640+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:26.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:26.640+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:26.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:26.640+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:26.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:26.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:26.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:26.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:26.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:26.567+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:26.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:26.567+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:27.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:26.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:27.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:26.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:27.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:26.929+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:27.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:26.929+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:27.569 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:06:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:06:27.200+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 48030 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:06:27.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:27.570+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:27.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:27.570+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:27.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:27.570+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:27.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:27.571+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:27.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:27.646+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:27.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:27.646+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:27.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:27.646+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:27.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:27.647+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:28.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:27.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:28.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:27.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:28.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:27.957+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:28.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:27.958+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:28.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:28.599+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:28.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:28.599+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:28.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:28.599+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:28.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:28.600+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:28.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:28.529+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:28.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:28.529+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:28.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:28.529+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:28.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:28.530+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:29.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:29.001+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:29.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:29.001+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:29.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:29.001+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:29.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:29.001+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:29.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:06:29.720 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:06:29.794 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:06:29.794 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-19T04:06:29.795 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-19T04:06:29.797 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609439/remote/smithi059/crash 2024-03-19T04:06:29.798 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/crash -- . 2024-03-19T04:06:29.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:29.543+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:29.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:29.543+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:29.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:29.543+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:29.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:29.543+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:29.823 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:29.594+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:29.823 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:29.594+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:29.824 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:29.594+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:29.824 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:29.595+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:29.826 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609439/remote/smithi176/crash 2024-03-19T04:06:29.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:06:29.828 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:06:30.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:30.007+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:30.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:30.007+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:30.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:30.007+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:30.008+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:30.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:30.568+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:30.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:30.568+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:30.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:30.568+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:30.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:30.569+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:30.577+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:30.577+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:30.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:30.577+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:30.840 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:30.578+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:31.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:31.026+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:31.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:31.026+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:31.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:31.026+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:31.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:31.027+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:31.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:31.529+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:31.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:31.529+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:31.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:31.529+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:31.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:31.530+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:31.822 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:31.582+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:31.822 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:31.582+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:31.822 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:31.582+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:31.823 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:31.583+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:32.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:32.025+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:32.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:32.025+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:32.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:32.025+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:32.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:32.026+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:32.514 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:06:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:06:32.207+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 48076 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:06:32.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:32.515+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:32.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:32.515+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:32.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:32.515+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:32.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:32.516+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:32.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:32.615+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:32.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:32.615+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:32.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:32.615+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:32.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:32.616+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:32.894 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:06:33.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:33.074+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:33.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:33.074+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:33.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:33.074+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:33.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:33.075+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:33.615 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:33.561+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:33.615 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:33.561+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:33.616 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:33.561+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:33.616 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:33.562+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:34.025 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:33.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:34.025 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:33.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:34.025 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:33.616+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:34.025 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:33.616+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:34.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:34.025+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:34.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:34.025+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:34.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:34.025+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:34.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:34.026+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:34.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:34.589+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:34.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:34.589+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:34.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:34.589+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:34.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:34.590+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:34.571+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:34.571+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:34.571+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:34.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:34.572+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:35.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:35.002+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:35.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:35.002+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:35.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:35.002+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:35.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:35.003+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:35.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:35.556+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:35.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:35.556+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:35.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:35.556+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:35.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:35.557+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:35.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:35.615+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:35.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:35.615+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:35.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:35.615+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:35.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:35.617+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:36.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:35.993+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:36.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:35.993+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:36.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:35.993+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:36.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:35.994+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:36.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:36.555+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:36.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:36.555+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:36.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:36.555+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:36.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:36.556+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:36.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:36.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:36.571+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:36.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:36.571+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:37.332 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:06:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:06:37.213+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 48122 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:06:37.333 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:37.016+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:37.333 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:37.016+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:37.333 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:37.016+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:37.333 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:37.016+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:37.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:37.578+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:37.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:37.578+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:37.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:37.578+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:37.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:37.579+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:37.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:37.544+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:37.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:37.544+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:37.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:37.544+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:37.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:37.545+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:38.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:38.008+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:38.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:38.008+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:38.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:38.008+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:38.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:38.009+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:38.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:38.616+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:38.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:38.616+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:38.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:38.616+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:38.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:38.617+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:38.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:38.498+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:38.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:38.498+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:38.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:38.498+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:38.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:38.499+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:39.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:39.031+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:39.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:39.031+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:39.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:39.031+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:39.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:39.031+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:39.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:39.507+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:39.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:39.507+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:39.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:39.507+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:39.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:39.508+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:39.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:39.652+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:39.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:39.652+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:39.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:39.652+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:39.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:39.653+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:40.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:40.080+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:40.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:40.080+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:40.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:40.080+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:40.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:40.081+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:40.692 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:40.529+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:40.693 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:40.529+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:40.693 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:40.529+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:40.693 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:40.530+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:40.693 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:40.694+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:40.693 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:40.694+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:40.694 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:40.694+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:41.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:40.695+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:41.520 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:41.126+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:41.520 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:41.126+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:41.520 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:41.126+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:41.520 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:41.127+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:41.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:41.701+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:41.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:41.701+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:41.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:41.701+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:41.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:41.702+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:41.823 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:41.521+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:41.823 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:41.521+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:41.823 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:41.521+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:41.823 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:41.522+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:41.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:06:41.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:06:42.488 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:06:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:06:42.220+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 48168 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:06:42.489 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:42.104+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:42.489 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:42.104+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:42.489 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:42.104+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:42.489 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:42.105+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:42.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:42.489+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:42.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:42.489+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:42.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:42.489+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:42.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:42.490+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:42.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:42.724+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:42.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:42.724+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:42.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:42.724+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:42.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:42.725+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:43.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:43.076+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:43.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:43.076+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:43.076+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:43.077+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:43.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:43.718+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:43.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:43.718+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:43.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:43.718+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:43.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:43.719+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:43.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:43.508+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:43.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:43.508+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:43.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:43.508+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:43.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:43.509+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:44.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:44.062+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:44.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:44.062+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:44.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:44.062+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:44.063+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:44.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:44.708+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:44.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:44.708+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:44.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:44.708+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:44.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:44.709+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:44.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:44.471+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:44.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:44.471+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:44.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:44.471+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:44.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:44.472+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:44.958 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:06:45.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:45.068+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:45.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:45.068+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:45.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:45.068+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:45.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:45.069+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:45.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:45.661+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:45.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:45.661+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:45.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:45.661+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:45.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:45.662+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:45.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:45.436+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:45.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:45.436+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:45.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:45.436+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:45.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:45.436+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:46.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:46.041+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:46.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:46.041+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:46.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:46.041+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:46.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:46.042+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:46.662 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:46.400+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:46.663 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:46.400+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:46.663 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:46.400+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:46.663 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:46.401+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:47.006 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:46.663+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:47.006 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:46.663+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:47.006 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:46.663+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:47.006 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:46.664+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:47.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:06:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:06:47.225+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 48215 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:06:47.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:47.006+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:47.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:47.006+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:47.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:47.006+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:47.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:47.007+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:47.679 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:47.386+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:47.679 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:47.386+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:47.680 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:47.386+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:47.680 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:47.387+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:47.980 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:47.679+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:47.980 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:47.679+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:47.980 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:47.679+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:47.980 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:47.680+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:48.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:47.980+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:48.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:47.980+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:48.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:47.980+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:48.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:47.981+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:48.673 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:48.410+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:48.674 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:48.410+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:48.674 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:48.410+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:48.674 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:48.411+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:49.000 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:48.674+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:49.000 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:48.674+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:49.001 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:48.674+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:49.001 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:48.675+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:49.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:49.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:49.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:49.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:49.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:49.000+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:49.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:49.001+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:49.663 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:49.389+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:49.664 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:49.389+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:49.664 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:49.389+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:49.664 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:49.390+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:49.955 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:49.664+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:49.955 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:49.664+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:49.955 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:49.664+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:49.955 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:49.665+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:50.286 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:49.955+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:50.287 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:49.955+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:50.287 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:49.955+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:50.287 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:49.956+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:50.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:50.414+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:50.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:50.414+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:50.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:50.414+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:50.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:50.415+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:51.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:50.936+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:51.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:50.936+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:51.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:50.936+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:51.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:50.936+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:50.710+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:50.710+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:50.710+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:51.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:50.710+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:51.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:51.676+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:51.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:51.676+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:51.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:51.676+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:51.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:51.677+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:51.824 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:51.449+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:51.824 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:51.449+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:51.824 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:51.449+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:51.824 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:51.450+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:52.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:51.925+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:52.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:51.925+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:52.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:51.925+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:52.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:51.926+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:52.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:06:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:06:52.231+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 48261 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:06:52.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:52.419+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:52.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:52.419+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:52.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:52.419+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:52.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:52.420+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:53.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:52.877+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:52.877+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:52.877+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:53.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:52.878+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:53.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:52.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:53.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:52.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:53.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:52.646+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:53.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:52.647+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:53.833 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:53.420+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:53.834 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:53.420+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:53.834 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:53.420+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:53.834 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:53.421+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:53.834 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:53.652+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:53.834 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:53.652+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:53.835 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:53.652+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:53.835 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:53.653+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:54.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:53.834+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:54.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:53.834+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:54.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:53.834+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:54.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:53.834+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:54.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:54.441+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:54.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:54.441+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:54.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:54.441+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:54.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:54.442+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:54.608+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:54.608+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:54.608+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:54.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:54.609+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:55.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:54.844+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:55.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:54.844+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:55.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:54.844+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:55.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:54.844+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:55.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:55.636+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:55.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:55.636+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:55.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:55.636+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:55.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:55.637+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:55.826 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:55.438+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:55.826 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:55.438+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:55.826 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:55.438+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:55.826 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:55.439+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:56.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:55.825+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:56.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:55.825+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:56.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:55.825+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:56.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:55.826+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:56.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:56.659+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:56.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:56.659+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:56.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:56.659+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:56.783 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:56.660+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:56.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:56.434+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:56.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:56.434+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:56.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:56.434+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:56.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:56.435+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:56.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:06:56.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:06:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:56.782+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:56.782+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:56.782+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:57.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:56.783+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:57.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:06:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:06:57.237+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 48299 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:06:57.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:57.394+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:57.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:57.394+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:57.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:57.394+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:57.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:57.394+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:58.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:57.673+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:58.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:57.673+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:58.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:57.673+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:58.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:57.674+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:58.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:57.832+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:58.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:57.832+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:58.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:57.832+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:58.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:57.833+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:58.703 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:58.409+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:58.703 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:58.409+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:58.703 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:58.409+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:58.703 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:58.410+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:06:59.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:58.846+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:59.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:58.846+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:59.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:58.846+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:06:59.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:58.847+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:59.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:58.703+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:59.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:58.703+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:59.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:58.703+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:06:59.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:58.704+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:06:59.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:59.433+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:59.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:59.433+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:59.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:59.433+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:06:59.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:06:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:06:59.434+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:00.034 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:07:00.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:59.669+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:00.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:59.669+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:00.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:59.669+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:00.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:06:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:06:59.670+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:00.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:59.839+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:00.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:59.839+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:00.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:59.839+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:00.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:06:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:06:59.839+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:00.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:00.409+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:00.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:00.409+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:00.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:00.409+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:00.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:00.410+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:01.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:00.873+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:01.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:00.873+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:01.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:00.873+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:01.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:00.874+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:01.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:00.663+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:01.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:00.663+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:01.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:00.663+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:01.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:00.664+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:01.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:01.658+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:01.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:01.658+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:01.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:01.658+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:01.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:01.659+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:01.824 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:01.410+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:01.825 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:01.410+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:01.825 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:01.410+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:01.825 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:01.412+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:02.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:01.886+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:02.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:01.886+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:02.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:01.886+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:02.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:01.887+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:02.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:07:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:07:02.244+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 48345 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:07:02.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:02.392+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:02.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:02.392+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:02.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:02.392+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:02.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:02.393+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:02.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:02.655+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:02.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:02.655+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:02.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:02.655+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:02.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:02.655+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:03.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:07:03.036 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:07:03.102 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:07:03.102 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-19T04:07:03.103 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-19T04:07:03.103 DEBUG:teuthology.orchestra.run.smithi059:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-19T04:07:03.141 INFO:tasks.cephadm:Compressing logs... 2024-03-19T04:07:03.142 DEBUG:teuthology.orchestra.run.smithi059:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-19T04:07:03.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:07:03.185 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:07:03.206 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:02.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:03.207 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:02.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:03.207 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:02.930+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:03.207 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:02.931+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:03.208 INFO:teuthology.orchestra.run.smithi059.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose --: No such file or directory /var/log/ceph/cephadm.log 2024-03-19T04:07:03.209 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-19T04:07:03.209 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-19T04:07:03.210 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-19T04:07:03.210 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-19T04:07:03.211 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-mds.cephfs.smithi059.hmylwa.log/var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-osd.0.log: 2024-03-19T04:07:03.211 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-mds.cephfs.smithi059.doeuxg.log 2024-03-19T04:07:03.213 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-mds.cephfs.smithi059.hmylwa.log: /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-mds.cephfs.smithi059.doeuxg.log: gzip -5 --verbose -- /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-19T04:07:03.214 INFO:teuthology.orchestra.run.smithi059.stderr: 89.8% -- replaced with /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-19T04:07:03.214 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose --/var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph.cephadm.log: /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph.log 2024-03-19T04:07:03.214 INFO:teuthology.orchestra.run.smithi059.stderr: 81.6% -- replaced with /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-19T04:07:03.214 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-19T04:07:03.215 INFO:teuthology.orchestra.run.smithi059.stderr: 86.0% -- replaced with /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-19T04:07:03.217 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-19T04:07:03.217 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-mon.smithi059.log 2024-03-19T04:07:03.218 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-volume.log: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-19T04:07:03.220 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-mon.smithi059.log: gzip -5 --verbose -- /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-mgr.smithi059.azpecc.log 2024-03-19T04:07:03.231 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-mgr.smithi059.azpecc.log: 90.8% -- replaced with /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-19T04:07:03.258 INFO:teuthology.orchestra.run.smithi059.stderr: 95.8% -- replaced with /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-mds.cephfs.smithi059.doeuxg.log.gz 2024-03-19T04:07:03.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:03.438+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:03.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:03.438+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:03.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:03.438+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:03.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:03.438+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:03.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:03.616+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:03.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:03.616+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:03.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:03.616+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:03.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:03.617+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:04.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:03.975+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:04.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:03.975+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:04.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:03.975+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:04.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:03.977+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:04.349 INFO:teuthology.orchestra.run.smithi059.stderr:gzip: /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-mgr.smithi059.azpecc.log: file size changed while zipping 2024-03-19T04:07:04.349 INFO:teuthology.orchestra.run.smithi059.stderr: 89.8% -- replaced with /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-mgr.smithi059.azpecc.log.gz 2024-03-19T04:07:04.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:04.617+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:04.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:04.617+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:04.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:04.617+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:04.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:04.618+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:04.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:04.406+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:04.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:04.406+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:04.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:04.406+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:04.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:04.408+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:05.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:04.994+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:05.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:04.994+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:05.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:04.994+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:05.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:04.995+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:05.657 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:05.386+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:05.657 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:05.386+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:05.657 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:05.386+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:05.657 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:05.387+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:06.004 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:05.656+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:06.005 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:05.656+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:06.005 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:05.656+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:06.005 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:05.658+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:06.005 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:06.003+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:06.005 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:06.003+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:06.006 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:06.003+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:06.006 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:06.004+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:06.238 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:07:06.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:06.371+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:06.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:06.371+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:06.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:06.371+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:06.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:06.372+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:07.026 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:06.632+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:07.027 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:06.632+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:07.027 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:06.632+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:07.027 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:06.633+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:07.027 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:07.024+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:07.028 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:07.024+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:07.028 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:07.024+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:07.028 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:07.026+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:07.051 INFO:teuthology.orchestra.run.smithi059.stderr:gzip: /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-mon.smithi059.log: file size changed while zipping 2024-03-19T04:07:07.052 INFO:teuthology.orchestra.run.smithi059.stderr: 96.7% -- replaced with /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-mon.smithi059.log.gz 2024-03-19T04:07:07.151 INFO:teuthology.orchestra.run.smithi059.stderr:gzip: /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-osd.2.log: file size changed while zipping 2024-03-19T04:07:07.151 INFO:teuthology.orchestra.run.smithi059.stderr: 90.8% -- replaced with /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-19T04:07:07.289 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:07:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:07:07.251+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 48392 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:07:07.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:07.378+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:07.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:07.378+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:07.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:07.378+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:07.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:07.379+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:08.041 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:07.620+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:08.041 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:07.620+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:08.042 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:07.620+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:08.042 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:07.621+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:08.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:08.041+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:08.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:08.041+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:08.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:08.041+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:08.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:08.043+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:08.659 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:08.338+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:08.660 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:08.338+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:08.660 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:08.338+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:08.660 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:08.339+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:09.052 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:08.659+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:09.052 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:08.659+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:09.052 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:08.659+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:09.053 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:08.660+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:09.053 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:09.052+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:09.053 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:09.052+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:09.053 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:09.052+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:09.317 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:09.054+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:09.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:09.317+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:09.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:09.317+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:09.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:09.317+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:09.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:09.318+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:10.019 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:09.676+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:10.019 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:09.676+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:10.020 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:09.676+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:10.020 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:09 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:09.677+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:10.289 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:10.019+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:10.289 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:10.019+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:10.290 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:10.019+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:10.290 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:10.020+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:10.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:10.367+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:10.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:10.367+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:10.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:10.367+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:10.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:10.368+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:10.883 INFO:teuthology.orchestra.run.smithi059.stderr:gzip: /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-osd.0.log: file size changed while zipping 2024-03-19T04:07:10.884 INFO:teuthology.orchestra.run.smithi059.stderr: 95.1% -- replaced with /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-19T04:07:10.984 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:10.657+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:10.985 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:10.657+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:10.985 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:10.657+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:10.985 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:10.658+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:11.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:10.985+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:11.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:10.985+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:11.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:10.985+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:11.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:10 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:10.986+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:11.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:11.345+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:11.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:11.345+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:11.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:11.345+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:11.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:11.346+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:11.996 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:11.700+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:11.997 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:11.700+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:11.997 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:11.700+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:11.997 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:11.701+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:12.258 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:11.997+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:12.258 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:11.997+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:12.259 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:11.997+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:12.259 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:11 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:11.998+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:12.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:07:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:07:12.257+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 48438 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:07:12.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:12.389+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:12.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:12.389+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:12.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:12.389+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:12.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:12.390+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:13.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:12.734+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:13.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:12.734+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:13.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:12.734+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:13.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:12.735+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:12.982+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:12.982+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:12.982+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:13.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:12 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:12.982+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:13.747 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:13.432+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:13.747 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:13.432+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:13.747 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:13.432+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:13.747 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:13.433+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:14.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:13.747+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:14.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:13.747+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:14.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:13.747+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:14.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:13.748+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:14.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:13.963+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:14.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:13.963+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:14.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:13.963+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:14.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:13 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:13.964+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:14.789 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:14.467+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:14.789 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:14.467+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:14.789 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:14.467+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:14.790 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:14.468+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:15.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:14.951+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:14.951+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:14.951+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:15.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:14.952+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:15.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:14.789+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:15.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:14.789+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:15.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:14.789+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:15.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:14 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:14.790+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:15.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:07:15.241 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:07:15.798 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:15.450+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:15.799 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:15.450+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:15.799 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:15.450+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:15.799 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:15.451+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:16.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:15.797+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:16.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:15.797+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:16.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:15.797+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:16.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:15.799+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:16.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:15.912+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:16.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:15.912+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:16.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:15.912+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:16.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:15 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:15.913+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:16.195 INFO:teuthology.orchestra.run.smithi059.stderr:gzip: /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-osd.1.log: file size changed while zipping 2024-03-19T04:07:16.196 INFO:teuthology.orchestra.run.smithi059.stderr: 96.5% -- replaced with /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-19T04:07:16.794 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:16.468+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:16.794 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:16.468+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:16.795 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:16.468+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:16.795 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:16.469+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:17.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:16.794+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:17.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:16.794+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:17.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:16.794+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:17.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:16.795+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:16.946+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:16.946+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:16.946+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:17.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:16 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:16.947+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:17.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:07:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:07:17.264+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 48484 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:07:17.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:17.429+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:17.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:17.429+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:17.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:17.429+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:17.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:17.429+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:18.306 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:07:18.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:17.977+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:18.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:17.977+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:17.977+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:18.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:17.978+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:18.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:17.842+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:18.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:17.842+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:18.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:17.842+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:18.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:17 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:17.843+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:18.820 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:18.447+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:18.821 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:18.447+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:18.821 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:18.447+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:18.821 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:18.448+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:19.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:18.980+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:19.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:18.980+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:19.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:18.980+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:19.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:18.981+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:19.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:18.821+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:19.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:18.821+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:19.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:18.821+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:19.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:18 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:18.822+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:19.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:19.424+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:19.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:19.424+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:19.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:19.424+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:19.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:19.424+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:20.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:19.803+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:20.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:19.803+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:20.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:19.803+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:20.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:19 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:19.804+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:20.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:20.002+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:20.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:20.002+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:20.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:20.002+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:20.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:20.002+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:20.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:20.406+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:20.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:20.406+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:20.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:20.406+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:20.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:20.407+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:21.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:20.808+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:21.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:20.808+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:21.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:20.808+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:21.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:20.808+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:21.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:20.973+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:21.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:20.973+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:21.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:20.973+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:21.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:20 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:20.973+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:21.776 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:21.374+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:21.777 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:21.374+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:21.777 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:21.374+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:21.777 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:21.375+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:22.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:21.777+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:22.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:21.777+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:22.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:21.777+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:22.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:21.778+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:22.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:21.984+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:22.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:21.984+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:22.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:21.984+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:22.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:21 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:21.985+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:22.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:07:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:07:22.269+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 48530 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:07:22.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:22.419+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:22.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:22.419+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:22.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:22.419+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:22.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:22.420+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:22.678 INFO:teuthology.orchestra.run.smithi059.stderr: 91.8% -- replaced with /var/log/ceph/8c8ae530-e59d-11ee-95c9-87774f69a715/ceph-mds.cephfs.smithi059.hmylwa.log.gz 2024-03-19T04:07:22.719 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-19T04:07:22.719 INFO:teuthology.orchestra.run.smithi059.stderr:real 0m19.485s 2024-03-19T04:07:22.719 INFO:teuthology.orchestra.run.smithi059.stderr:user 0m47.858s 2024-03-19T04:07:22.719 INFO:teuthology.orchestra.run.smithi059.stderr:sys 0m0.991s 2024-03-19T04:07:23.030 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:22.815+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:23.030 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:22.815+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:23.030 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:22.815+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:23.031 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:22 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:22.816+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:23.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:23.030+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:23.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:23.030+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:23.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:23.030+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:23.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:23.031+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:23.782 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:23.427+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:23.782 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:23.427+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:23.783 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:23.427+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:23.783 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:23.428+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:24.045 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:23.782+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:24.046 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:23.782+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:24.046 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:23.782+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:24.046 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:23 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:23.783+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:24.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:24.045+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:24.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:24.045+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:24.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:24.045+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:24.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:24.046+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:24.816 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:24.422+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:24.816 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:24.422+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:24.816 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:24.422+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:24.816 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:24.423+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:25.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:24.816+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:25.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:24.816+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:25.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:24.816+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:25.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:24 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:24.817+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:25.058+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:25.058+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:25.058+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:25.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:25.059+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:25.827 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:25.393+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:25.828 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:25.393+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:25.828 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:25.393+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:25.828 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:25.394+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:26.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:25.827+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:26.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:25.827+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:26.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:25.827+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:26.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:25 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:25.829+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:26.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:26.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:26.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:26.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:26.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:26.057+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:26.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:26.057+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:26.794 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:26.399+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:26.795 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:26.399+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:26.795 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:26.399+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:26.795 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:26.400+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:27.059 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:26.795+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:27.059 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:26.795+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:27.059 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:26.795+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:27.059 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:26 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:26.796+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:27.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:07:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:07:27.275+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 48577 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:07:27.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:27.059+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:27.059+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:27.059+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:27.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:27.059+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:27.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:27.380+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:27.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:27.380+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:27.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:27.380+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:27.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:27.381+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:28.337 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:27.837+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:28.337 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:27.837+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:28.337 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:27.837+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:28.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:27 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:27.838+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:28.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:28.028+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:28.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:28.028+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:28.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:28.028+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:28.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:28.029+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:28.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:28.340+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:28.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:28.340+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:28.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:28.340+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:28.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:28.341+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:29.244 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:28.846+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:29.244 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:28.846+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:29.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:28.846+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:29.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:28 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:28.847+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:29.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:29.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:29.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:29.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:29.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:29.032+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:29.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:29.033+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:29.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:29.337+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:29.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:29.337+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:29.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:29.337+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:29.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:29.338+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:30.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:29.809+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:30.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:29.809+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:30.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:29.809+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:30.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:29 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:29.810+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:30.041+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:30.041+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:30.041+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:30.088 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:30.042+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:30.307 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:07:30.308 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:07:30.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:30.386+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:30.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:30.386+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:30.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:30.386+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:30.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:30.387+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:31.073 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:30.811+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:31.074 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:30.811+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:31.074 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:30.811+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:31.074 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:30 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:30.812+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:31.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:31.073+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:31.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:31.073+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:31.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:31.073+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:31.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:31.075+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:31.799 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:31.401+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:31.799 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:31.401+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:31.799 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:31.401+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:31.799 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:31.402+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:32.065 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:31.799+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:32.065 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:31.799+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:32.065 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:31.799+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:32.065 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:31 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:31.800+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:32.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:07:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:07:32.282+0000 7f425eef9700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 48623 slow ops, oldest is log(1 entries from seq 288 at 2024-03-19T03:15:10.986225+0000) 2024-03-19T04:07:32.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:32.065+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:32.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:32.065+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:32.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:32.065+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:32.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:32.066+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:32.642 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:32.362+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:32.643 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:32.362+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:32.643 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:32.362+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:32.643 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:32.363+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:33.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:32.774+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:33.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:32.774+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:33.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:32.774+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:33.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:32 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:32.775+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:33.374 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:07:33.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:33.336+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:33.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:33.336+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:33.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:33.336+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:33.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:33.336+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:33.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:33.102+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:33.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:33.102+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:33.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:33.102+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:33.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:33.103+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:34.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:33.771+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:34.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:33.771+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:34.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:33.771+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:34.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:33 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:33.772+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:34.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:34.322+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:34.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:34.322+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:34.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:34.322+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:34.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:34.323+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:34.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:34.113+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:34.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:34.113+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:34.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:34.113+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:34.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:34.114+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:35.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:34.729+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:35.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:34.729+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:35.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:34.729+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:35.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:34 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:34.731+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:35.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:35.341+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:35.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:35.341+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:35.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:35.341+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:35.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:35.343+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:35.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:35.158+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:35.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:35.158+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:35.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:35.158+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:35.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:35.159+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:36.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:35.684+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:36.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:35.684+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:36.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:35.684+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:36.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:35 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:35.685+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:36.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:07:36.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:07:36.450 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:07:36.450 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-19T04:07:36.452 INFO:tasks.cephadm:Removing cluster... 2024-03-19T04:07:36.452 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 --force 2024-03-19T04:07:36.477 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:36.319+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:36.477 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:36.319+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:36.477 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:36.319+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:36.478 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:36.320+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:36.478 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:36.160+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:36.478 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:36.160+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:36.478 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:36.160+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:36.478 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:36.161+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:36.746 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:36.659+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:36.747 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:36.659+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:36.747 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:36.659+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:36.747 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:36 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:36.660+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:36.747 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:07:36 smithi059 systemd[1]: Stopping Ceph mon.smithi059 for 8c8ae530-e59d-11ee-95c9-87774f69a715... 2024-03-19T04:07:37.328 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:07:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:07:37.017+0000 7f4261eff700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi059 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T04:07:37.329 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:07:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059[29129]: 2024-03-19T04:07:37.017+0000 7f4261eff700 -1 mon.smithi059@0(probing) e2 *** Got Signal Terminated *** 2024-03-19T04:07:37.329 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:37.205+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:37.329 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:37.205+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:37.330 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:37.205+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:37.330 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:37.206+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:37.584 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:37.329+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:37.585 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:37.329+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:37.585 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:37.329+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:37.585 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:37.330+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:37.837 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:07:37 smithi059 podman[87025]: 2024-03-19 04:07:37.585903428 +0000 UTC m=+0.688908050 container died c8bc018eaa37f1af8b44320ee4ad903acd05b4c8deca961bf0ad1bf7e62df07d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=quincy-b4843bd, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-19T04:07:37.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:37.650+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:37.651+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:37.651+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:37.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:37 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:37.652+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:38.217 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:07:38 smithi059 podman[87025]: 2024-03-19 04:07:38.076965509 +0000 UTC m=+1.179970129 container cleanup c8bc018eaa37f1af8b44320ee4ad903acd05b4c8deca961bf0ad1bf7e62df07d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=quincy-b4843bd, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True) 2024-03-19T04:07:38.217 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:07:38 smithi059 bash[87025]: ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059 2024-03-19T04:07:38.493 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:07:38 smithi059 podman[87039]: 2024-03-19 04:07:38.362029914 +0000 UTC m=+0.772415911 container remove c8bc018eaa37f1af8b44320ee4ad903acd05b4c8deca961bf0ad1bf7e62df07d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-mon-smithi059, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=quincy-b4843bd, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-19T04:07:38.494 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:38.217+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:38.494 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:38.217+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:38.494 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:38.217+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:38.494 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:38.218+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:38.494 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:38.317+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:38.495 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:38.317+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:38.495 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:38.317+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:38.495 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:38.318+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:38.837 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:07:38 smithi059 systemd[1]: ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@mon.smithi059.service: Deactivated successfully. 2024-03-19T04:07:38.837 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:07:38 smithi059 systemd[1]: Stopped Ceph mon.smithi059 for 8c8ae530-e59d-11ee-95c9-87774f69a715. 2024-03-19T04:07:38.838 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Mar 19 04:07:38 smithi059 systemd[1]: ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@mon.smithi059.service: Consumed 11min 54.550s CPU time. 2024-03-19T04:07:38.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:38.685+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:38.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:38.685+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:38.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:38.685+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:38.839 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:38 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:38.686+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:39.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:39.273+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:39.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:39.273+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:39.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:39.273+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:39.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:39.274+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:39.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:39.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:39.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:39.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:39.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:39.239+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:39.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:39.239+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:39.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:39.714+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:39.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:39.714+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:39.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:39.714+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:39.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:39 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:39.715+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:40.485 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:40.308+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:40.486 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:40.308+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:40.486 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:40.308+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:40.486 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:40.309+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:40.486 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:40.215+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:40.486 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:40.215+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:40.487 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:40.215+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:40.487 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:40.215+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:40.779 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:40.697+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:40.779 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:40.697+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:40.779 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:40.697+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:40.779 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:40 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:40.698+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:41.500 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:41.295+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:41.501 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:41.295+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:41.501 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:41.295+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:41.501 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:41.296+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:41.501 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:41.184+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:41.501 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:41.184+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:41.501 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:41.184+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:41.501 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:41.184+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:41.784 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:41.658+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:41.784 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:41.658+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:41.784 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:41.658+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:41.784 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:41 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:41.659+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:42.333 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:42.145+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:42.333 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:42.145+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:42.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:42.145+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:42.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:42.146+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:42.584 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:42.333+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:42.584 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:42.333+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:42.585 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:42.333+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:42.585 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:42.334+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:43.053 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:42.657+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:43.053 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:42.657+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:43.054 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:42.657+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:43.054 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:42 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:42.657+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:43.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:43.125+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:43.125+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:43.125+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:43.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:43.126+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:43.640 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:43.365+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:43.641 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:43.365+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:43.641 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:43.365+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:43.641 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:43.366+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:43.990 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:43.640+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:43.990 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:43.640+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:43.990 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:43.640+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:43.990 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:43 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:43.641+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:44.320 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:44.129+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:44.321 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:44.129+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:44.321 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:44.129+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:44.321 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:44.130+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:44.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:44.320+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:44.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:44.320+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:44.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:44.320+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:44.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:44.321+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:45.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:44.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:45.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:44.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:45.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:44.660+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:45.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:44 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:44.661+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:45.461 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:45.153+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:45.462 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:45.153+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:45.462 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:45.153+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:45.462 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:45.153+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:45.462 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:45.279+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:45.462 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:45.279+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:45.462 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:45.279+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:45.463 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:45.281+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:45.739 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:45.655+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:45.739 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:45.655+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:45.739 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:45.655+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:45.739 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:45 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:45.656+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:46.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:46.320+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:46.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:46.320+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:46.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:46.320+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:46.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:46.321+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:46.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:46.135+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:46.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:46.135+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:46.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:46.135+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:46.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:46.136+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:47.051 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:46.651+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:47.052 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:46.651+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:47.052 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:46.651+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:47.052 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:46 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:46.651+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:47.308 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:47 smithi059 systemd[1]: Stopping Ceph osd.0 for 8c8ae530-e59d-11ee-95c9-87774f69a715... 2024-03-19T04:07:47.308 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:47.125+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:47.308 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:47.125+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:47.309 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:47.125+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:47.309 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:47.126+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:47.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:47.320+0000 7f57fd2ea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T04:07:47.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:47.320+0000 7f57fd2ea700 -1 osd.0 45 *** Got signal Terminated *** 2024-03-19T04:07:47.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:47.320+0000 7f57fd2ea700 -1 osd.0 45 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T04:07:47.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:47.308+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:47.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:47.308+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:47.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:47.308+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:47.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:47.309+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:48.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:47.612+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:48.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:47.612+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:48.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:47.612+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:48.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:47 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:47.613+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:48.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:48.135+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:48.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:48.135+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:48.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:48.135+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:48.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:48.136+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:48.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:48.290+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:48.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:48.290+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:48.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:48.290+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:48.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:48.291+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:49.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:48.596+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:49.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:48.596+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:49.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:48.596+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:49.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:48 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:48.596+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:49.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:49.293+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:49.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:49.293+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:49.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:49.293+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:49.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:49.294+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:49.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:49.172+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:49.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:49.172+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:49.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:49.172+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:49.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:49.173+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:49.638+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:49.638+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:49.638+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:50.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:49 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:49.639+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:50.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:50.128+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:50.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:50.128+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:50.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:50.128+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:50.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:50.128+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:50.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:50.294+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:50.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:50.294+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:50.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:50.294+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:50.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:50.295+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:51.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:50.635+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:50.635+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:50.635+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:51.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:50 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:50.636+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:51.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:51.335+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:51.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:51.335+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:51.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:51.335+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:51.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:51.335+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:51.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:51.157+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:51.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:51.157+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:51.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:51.157+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:51.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:51.157+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:52.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:51.626+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:52.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:51.626+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:52.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:51.626+0000 7f57f98e8700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:37.925472+0000 (oldest deadline 2024-03-19T03:57:57.925472+0000) 2024-03-19T04:07:52.088 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:51 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0[47604]: 2024-03-19T04:07:51.626+0000 7f57f98e8700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:4225 3.17 3:e92f07cd:::10000003c7c.00000000:head [write 0~1180 in=1180b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:52.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:52.287+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:52.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:52.287+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:52.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:52.287+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:52.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:52.287+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:52.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:52.147+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:52.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:52.147+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:52.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:52.147+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:52.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:52 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:52.148+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:53.194 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:52 smithi059 podman[87861]: 2024-03-19 04:07:52.902250387 +0000 UTC m=+5.701017458 container died 2511279e88524919674ea6b189c93a6b524af0ec43643ea1357b7b6d22352985 (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=quincy-b4843bd) 2024-03-19T04:07:53.554 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:53 smithi059 podman[87861]: 2024-03-19 04:07:53.287971548 +0000 UTC m=+6.086738692 container cleanup 2511279e88524919674ea6b189c93a6b524af0ec43643ea1357b7b6d22352985 (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=quincy-b4843bd, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-19T04:07:53.555 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:53 smithi059 bash[87861]: ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0 2024-03-19T04:07:53.555 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:53.270+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:53.555 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:53.270+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:53.555 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:53.270+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:53.556 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:53.271+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:53.556 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:53.194+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:53.556 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:53.194+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:53.556 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:53.194+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:53.556 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:53 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:53.195+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:53.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:53 smithi059 podman[87881]: 2024-03-19 04:07:53.555989824 +0000 UTC m=+0.649248152 container remove 2511279e88524919674ea6b189c93a6b524af0ec43643ea1357b7b6d22352985 (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=quincy-b4843bd) 2024-03-19T04:07:54.216 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:53 smithi059 podman[87971]: 2024-03-19 04:07:53.820748122 +0000 UTC m=+0.018057440 image pull 810955603526f1a5056294fef84995d43fccb8e7018e3b8031db5047002886ab quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881 2024-03-19T04:07:54.217 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:53 smithi059 podman[87971]: 2024-03-19 04:07:53.957008987 +0000 UTC m=+0.154318327 container create 44c920b9112edff69d2acbc121e6f1287c1f16a8764c2b8091d49b676e8ebada (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-b4843bd, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-03-19T04:07:54.217 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:54 smithi059 podman[87971]: 2024-03-19 04:07:54.217769228 +0000 UTC m=+0.415078526 container init 44c920b9112edff69d2acbc121e6f1287c1f16a8764c2b8091d49b676e8ebada (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=quincy-b4843bd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T04:07:54.217 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:54.157+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:54.217 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:54.157+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:54.217 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:54.158+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:54.217 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:54.158+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:54.528 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:54 smithi059 podman[87971]: 2024-03-19 04:07:54.221028713 +0000 UTC m=+0.418338020 container start 44c920b9112edff69d2acbc121e6f1287c1f16a8764c2b8091d49b676e8ebada (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=quincy-b4843bd) 2024-03-19T04:07:54.528 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:54 smithi059 podman[87971]: 2024-03-19 04:07:54.317954274 +0000 UTC m=+0.515263573 container attach 44c920b9112edff69d2acbc121e6f1287c1f16a8764c2b8091d49b676e8ebada (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, RELEASE=quincy-b4843bd) 2024-03-19T04:07:54.529 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:54 smithi059 podman[87971]: 2024-03-19 04:07:54.529845552 +0000 UTC m=+0.727154863 container died 44c920b9112edff69d2acbc121e6f1287c1f16a8764c2b8091d49b676e8ebada (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=quincy-b4843bd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-19T04:07:54.529 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:54.277+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:54.529 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:54.277+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:54.529 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:54.277+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:54.529 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:54 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:54.279+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:55.390 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:55 smithi059 podman[87997]: 2024-03-19 04:07:55.163833218 +0000 UTC m=+0.628142809 container remove 44c920b9112edff69d2acbc121e6f1287c1f16a8764c2b8091d49b676e8ebada (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=quincy-b4843bd, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T04:07:55.390 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:55 smithi059 systemd[1]: ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-19T04:07:55.390 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:55 smithi059 systemd[1]: Stopped Ceph osd.0 for 8c8ae530-e59d-11ee-95c9-87774f69a715. 2024-03-19T04:07:55.390 INFO:journalctl@ceph.osd.0.smithi059.stdout:Mar 19 04:07:55 smithi059 systemd[1]: ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@osd.0.service: Consumed 25min 43.586s CPU time. 2024-03-19T04:07:55.391 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:55.262+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:55.391 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:55.262+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:55.391 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:55.262+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:55.391 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:55.263+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:55.392 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:55.116+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:55.392 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:55.116+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:55.392 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:55.116+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:55.392 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:55.117+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:55.649 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:55 smithi059 systemd[1]: Stopping Ceph osd.1 for 8c8ae530-e59d-11ee-95c9-87774f69a715... 2024-03-19T04:07:56.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:55.658+0000 7fc4a27f7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T04:07:56.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:55.658+0000 7fc4a27f7700 -1 osd.1 45 *** Got signal Terminated *** 2024-03-19T04:07:56.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:55 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:55.658+0000 7fc4a27f7700 -1 osd.1 45 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T04:07:56.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:56.242+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:56.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:56.242+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:56.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:56.242+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:56.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:56.243+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:56.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:56.125+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:56.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:56.125+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:56.125+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:56.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:56 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:56.126+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:57.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:57.202+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:57.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:57.202+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:57.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:57.202+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:57.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:57.202+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:57.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:57.150+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:57.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:57.150+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:57.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:57.150+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:57.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:57 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:57.150+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:58.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:58.147+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:58.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:58.147+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:58.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:58.147+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:58.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:58.147+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:07:58.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:58.190+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:58.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:58.190+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:58.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:58.190+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:58.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:58 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:58.190+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:59.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:59.206+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:59.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:59.206+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:59.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:59.206+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:07:59.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:07:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:07:59.206+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:07:59.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:59.127+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:59.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:59.127+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:59.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:59.127+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:07:59.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:07:59 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:07:59.128+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:08:00.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:08:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:08:00.201+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:08:00.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:08:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:08:00.201+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:08:00.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:08:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:08:00.201+0000 7fc49c5f0700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:28.778725+0000 (oldest deadline 2024-03-19T03:57:48.778725+0000) 2024-03-19T04:08:00.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:08:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1[54966]: 2024-03-19T04:08:00.202+0000 7fc49c5f0700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:374 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=263846b,omap-rm-keys in=49b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-03-19T04:08:00.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:00.133+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:08:00.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:00.133+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:08:00.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:00.133+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:08:00.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:00 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:00.134+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:08:01.525 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:08:01 smithi059 podman[88084]: 2024-03-19 04:08:01.255543124 +0000 UTC m=+5.707867869 container died cd85f348d650d20be74e79dcceb584a8c3fccf8da5c14329da3b72226c5feb54 (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=quincy-b4843bd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T04:08:01.525 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:01.119+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:08:01.525 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:01.119+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:08:01.525 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:01.119+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:08:01.526 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:01 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:01.120+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:08:01.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:08:01 smithi059 podman[88084]: 2024-03-19 04:08:01.600687051 +0000 UTC m=+6.053011804 container cleanup cd85f348d650d20be74e79dcceb584a8c3fccf8da5c14329da3b72226c5feb54 (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=quincy-b4843bd, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-19T04:08:01.838 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:08:01 smithi059 bash[88084]: ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1 2024-03-19T04:08:02.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:08:01 smithi059 podman[88102]: 2024-03-19 04:08:01.885483467 +0000 UTC m=+0.631412128 container remove cd85f348d650d20be74e79dcceb584a8c3fccf8da5c14329da3b72226c5feb54 (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=quincy-b4843bd, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T04:08:02.214 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:02.103+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:08:02.214 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:02.103+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:08:02.214 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:02.103+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:08:02.214 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:02 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:02.104+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:08:02.504 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:08:02 smithi059 podman[88192]: 2024-03-19 04:08:02.118862347 +0000 UTC m=+0.017931185 image pull 810955603526f1a5056294fef84995d43fccb8e7018e3b8031db5047002886ab quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881 2024-03-19T04:08:02.504 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:08:02 smithi059 podman[88192]: 2024-03-19 04:08:02.219481135 +0000 UTC m=+0.118549965 container create 850d22577878ca3071ab45b2ce90ee2fae9f00e02947fcf2e7d9dba791ac531c (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=quincy-b4843bd, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True) 2024-03-19T04:08:02.504 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:08:02 smithi059 podman[88192]: 2024-03-19 04:08:02.421930743 +0000 UTC m=+0.320999582 container init 850d22577878ca3071ab45b2ce90ee2fae9f00e02947fcf2e7d9dba791ac531c (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=quincy-b4843bd, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-19T04:08:02.504 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:08:02 smithi059 podman[88192]: 2024-03-19 04:08:02.424924884 +0000 UTC m=+0.323993723 container start 850d22577878ca3071ab45b2ce90ee2fae9f00e02947fcf2e7d9dba791ac531c (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=quincy-b4843bd, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T04:08:02.504 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:08:02 smithi059 podman[88192]: 2024-03-19 04:08:02.505135603 +0000 UTC m=+0.404204437 container attach 850d22577878ca3071ab45b2ce90ee2fae9f00e02947fcf2e7d9dba791ac531c (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=quincy-b4843bd, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True) 2024-03-19T04:08:02.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:08:02 smithi059 podman[88192]: 2024-03-19 04:08:02.712980107 +0000 UTC m=+0.612048941 container died 850d22577878ca3071ab45b2ce90ee2fae9f00e02947fcf2e7d9dba791ac531c (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, RELEASE=quincy-b4843bd) 2024-03-19T04:08:03.248 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:03.116+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6804 osd.3 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:08:03.249 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:03.116+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6812 osd.4 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:08:03.249 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:03.116+0000 7fa7db3d0700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.176:6820 osd.5 ever on either front or back, first ping sent 2024-03-19T03:57:29.095695+0000 (oldest deadline 2024-03-19T03:57:49.095695+0000) 2024-03-19T04:08:03.249 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:03.117+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:08:03.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:03 smithi059 systemd[1]: Stopping Ceph osd.2 for 8c8ae530-e59d-11ee-95c9-87774f69a715... 2024-03-19T04:08:03.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:08:03 smithi059 podman[88218]: 2024-03-19 04:08:03.250034589 +0000 UTC m=+0.530158957 container remove 850d22577878ca3071ab45b2ce90ee2fae9f00e02947fcf2e7d9dba791ac531c (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=quincy-b4843bd, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-19T04:08:03.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:08:03 smithi059 systemd[1]: ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-19T04:08:03.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:08:03 smithi059 systemd[1]: Stopped Ceph osd.1 for 8c8ae530-e59d-11ee-95c9-87774f69a715. 2024-03-19T04:08:03.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Mar 19 04:08:03 smithi059 systemd[1]: ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@osd.1.service: Consumed 50min 45.196s CPU time. 2024-03-19T04:08:04.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:03.703+0000 7fa7e1dd8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T04:08:04.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:03.703+0000 7fa7e1dd8700 -1 osd.2 45 *** Got signal Terminated *** 2024-03-19T04:08:04.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:03 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:03.703+0000 7fa7e1dd8700 -1 osd.2 45 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T04:08:04.566 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:04 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:04.158+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:08:05.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:05 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:05.183+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:08:06.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:06 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:06.200+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:08:07.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:07 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:07.168+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:08:08.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:08 smithi059 ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2[62290]: 2024-03-19T04:08:08.193+0000 7fa7db3d0700 -1 osd.2 45 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.14530.0:4232 3.7 3:e7b12498:::10000003c7e.00000000:head [write 0~2130 in=2130b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-03-19T04:08:09.418 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:09 smithi059 podman[88296]: 2024-03-19 04:08:09.112224571 +0000 UTC m=+5.507041309 container died 363582327fdecaa371ed6215d4b7587fccbcd689be432da50fe03a308c49f79f (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=quincy-b4843bd, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T04:08:09.418 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:09 smithi059 podman[88296]: 2024-03-19 04:08:09.419733874 +0000 UTC m=+5.814550600 container cleanup 363582327fdecaa371ed6215d4b7587fccbcd689be432da50fe03a308c49f79f (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=quincy-b4843bd, maintainer=Guillaume Abrioux ) 2024-03-19T04:08:09.837 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:09 smithi059 bash[88296]: ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2 2024-03-19T04:08:09.837 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:09 smithi059 podman[88315]: 2024-03-19 04:08:09.595728931 +0000 UTC m=+0.481324487 container remove 363582327fdecaa371ed6215d4b7587fccbcd689be432da50fe03a308c49f79f (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=quincy-b4843bd, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T04:08:10.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:09 smithi059 podman[88411]: 2024-03-19 04:08:09.812467629 +0000 UTC m=+0.017609927 image pull 810955603526f1a5056294fef84995d43fccb8e7018e3b8031db5047002886ab quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881 2024-03-19T04:08:10.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:09 smithi059 podman[88411]: 2024-03-19 04:08:09.93835769 +0000 UTC m=+0.143500010 container create 666aee6c99d50623341e0af5dc6a7fd0043f22171f724b62bf2494ec0b53ac9b (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=quincy-b4843bd, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T04:08:10.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:10 smithi059 podman[88411]: 2024-03-19 04:08:10.140669739 +0000 UTC m=+0.345812031 container init 666aee6c99d50623341e0af5dc6a7fd0043f22171f724b62bf2494ec0b53ac9b (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=quincy-b4843bd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-19T04:08:10.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:10 smithi059 podman[88411]: 2024-03-19 04:08:10.143641775 +0000 UTC m=+0.348784061 container start 666aee6c99d50623341e0af5dc6a7fd0043f22171f724b62bf2494ec0b53ac9b (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=quincy-b4843bd, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T04:08:10.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:10 smithi059 podman[88411]: 2024-03-19 04:08:10.223785879 +0000 UTC m=+0.428928158 container attach 666aee6c99d50623341e0af5dc6a7fd0043f22171f724b62bf2494ec0b53ac9b (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=quincy-b4843bd, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-19T04:08:10.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:10 smithi059 podman[88411]: 2024-03-19 04:08:10.432349418 +0000 UTC m=+0.637491709 container died 666aee6c99d50623341e0af5dc6a7fd0043f22171f724b62bf2494ec0b53ac9b (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=quincy-b4843bd, maintainer=Guillaume Abrioux ) 2024-03-19T04:08:10.990 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:10 smithi059 podman[88437]: 2024-03-19 04:08:10.952184194 +0000 UTC m=+0.514854723 container remove 666aee6c99d50623341e0af5dc6a7fd0043f22171f724b62bf2494ec0b53ac9b (image=quay.ceph.io/ceph-ci/ceph@sha256:fcce6e5ef923cc5e23f5bcf3a2b7451ad8212e9ac39077620fa2d1e53b4d6881, name=ceph-8c8ae530-e59d-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=quincy-b4843bd, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-19T04:08:10.991 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:10 smithi059 systemd[1]: ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-19T04:08:10.991 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:10 smithi059 systemd[1]: Stopped Ceph osd.2 for 8c8ae530-e59d-11ee-95c9-87774f69a715. 2024-03-19T04:08:10.991 INFO:journalctl@ceph.osd.2.smithi059.stdout:Mar 19 04:08:10 smithi059 systemd[1]: ceph-8c8ae530-e59d-11ee-95c9-87774f69a715@osd.2.service: Consumed 24.999s CPU time. 2024-03-19T04:08:16.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:08:16.274 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:08:19.326 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:08:28.327 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:08:28.328 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:08:31.390 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:08:43.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:08:43.393 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:08:46.462 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:08:49.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:08:49.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:08:49.534 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:08:49.534 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-19T04:08:49.535 INFO:tasks.cephadm:Teardown complete 2024-03-19T04:08:49.535 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephadm.py", line 1613, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephadm.py", line 1867, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephadm.py", line 1870, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephadm.py", line 1746, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephadm.py", line 200, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephadm.py", line 368, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi176 2024-03-19T04:08:49.564 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-19T04:08:49.578 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-03-19T04:08:49.596 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-19T04:08:49.596 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-19T04:08:49.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:08:49.609 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:08:52.670 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:09:01.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:09:01.672 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:09:04.734 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:09:16.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:09:16.736 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:09:19.806 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:09:22.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:09:22.808 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:09:22.878 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:09:22.878 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-19T04:09:22.928 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi176 2024-03-19T04:09:22.947 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-19T04:09:22.959 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-19T04:09:22.960 DEBUG:teuthology.orchestra.run.smithi059:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T04:09:22.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:09:22.968 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:09:22.983 INFO:teuthology.orchestra.run.smithi059.stderr:bash: line 1: ntpq: command not found 2024-03-19T04:09:22.988 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T04:09:22.988 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2024-03-19T04:09:22.989 INFO:teuthology.orchestra.run.smithi059.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 37 +3623us[+3623us] +/- 116ms 2024-03-19T04:09:22.989 INFO:teuthology.orchestra.run.smithi059.stdout:^* hv02.front.sepia.ceph.com 3 8 377 231 -1413us[-1552us] +/- 56ms 2024-03-19T04:09:22.989 INFO:teuthology.orchestra.run.smithi059.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 231 -612us[ -612us] +/- 79ms 2024-03-19T04:09:22.989 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T04:09:26.054 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:09:35.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:09:35.055 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:09:38.110 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:09:50.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:09:50.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:09:53.182 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:09:56.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:09:56.184 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:09:56.254 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:09:56.254 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-19T04:09:56.255 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi176 2024-03-19T04:09:56.270 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-19T04:09:56.282 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-19T04:09:56.283 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-19T04:09:56.314 DEBUG:teuthology.orchestra.run.smithi059:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-19T04:09:56.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:09:56.403 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:09:59.454 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:10:08.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:10:08.456 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:10:11.518 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:10:23.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:10:23.520 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:10:26.590 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:10:29.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:10:29.592 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:10:29.662 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:10:29.662 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-19T04:10:29.663 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi176 2024-03-19T04:10:29.672 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-19T04:10:29.685 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-19T04:10:29.713 INFO:teuthology.task.internal:Duration was 5745.039199 seconds 2024-03-19T04:10:29.714 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-19T04:10:29.730 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-19T04:10:29.731 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-19T04:10:29.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:10:29.737 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:10:29.786 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T04:10:32.802 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:10:41.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:10:41.803 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:10:44.862 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:10:56.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:10:56.864 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:10:59.934 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:11:02.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:11:02.935 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:11:03.006 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:11:03.006 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-19T04:11:03.007 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi176 2024-03-19T04:11:03.023 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-19T04:11:03.037 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-19T04:11:03.037 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-19T04:11:03.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:11:03.075 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:11:06.142 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:11:15.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:11:15.144 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:11:18.206 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:11:30.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:11:30.209 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:11:33.278 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:11:36.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:11:36.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:11:36.354 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:11:36.355 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-19T04:11:36.356 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi176 2024-03-19T04:11:36.374 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-19T04:11:36.388 DEBUG:teuthology.orchestra.run.smithi059:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-19T04:11:36.395 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:11:36.396 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:11:36.422 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = core 2024-03-19T04:11:39.454 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:11:48.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:11:48.456 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:11:51.518 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:12:03.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:12:03.525 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:12:06.590 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:12:09.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:12:09.592 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:12:09.666 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:12:09.666 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-19T04:12:09.667 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi176 2024-03-19T04:12:09.685 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-19T04:12:09.699 INFO:teuthology.task.internal:Transferring archived files... 2024-03-19T04:12:09.701 DEBUG:teuthology.misc:Transferring archived files from smithi059:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609439/remote/smithi059 2024-03-19T04:12:09.701 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-19T04:12:09.744 DEBUG:teuthology.misc:Transferring archived files from smithi176:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609439/remote/smithi176 2024-03-19T04:12:09.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:12:09.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:12:12.802 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:12:21.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:12:21.804 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:12:24.869 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:12:36.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:12:36.871 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:12:39.938 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:12:42.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:12:42.940 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:12:43.010 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:12:43.010 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-19T04:12:43.011 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi176 2024-03-19T04:12:43.029 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-19T04:12:43.041 INFO:teuthology.task.internal:Not uploading archives. 2024-03-19T04:12:43.041 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-19T04:12:43.097 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-19T04:12:43.098 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-19T04:12:43.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:12:43.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:12:43.121 INFO:teuthology.orchestra.run.smithi059.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 19 04:08 /home/ubuntu/cephtest 2024-03-19T04:12:43.121 INFO:teuthology.orchestra.run.smithi059.stdout: 262167 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 19 04:11 /home/ubuntu/cephtest/archive 2024-03-19T04:12:43.122 INFO:teuthology.orchestra.run.smithi059.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 02:34 /home/ubuntu/cephtest/archive/syslog 2024-03-19T04:12:43.122 INFO:teuthology.orchestra.run.smithi059.stdout: 265306 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Mar 19 02:34 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T04:12:43.122 INFO:teuthology.orchestra.run.smithi059.stdout: 265305 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Mar 19 02:34 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T04:12:43.122 INFO:teuthology.orchestra.run.smithi059.stdout: 402360 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 04:09 /home/ubuntu/cephtest/archive/audit 2024-03-19T04:12:43.122 INFO:teuthology.orchestra.run.smithi059.stdout: 426278 176 -rw------- 1 ubuntu ubuntu 178926 Mar 19 04:09 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-03-19T04:12:43.122 INFO:teuthology.orchestra.run.smithi059.stdout: 265395 368 -rwxr-xr-x 1 ubuntu ubuntu 375153 Mar 19 03:05 /home/ubuntu/cephtest/cephadm 2024-03-19T04:12:43.123 INFO:teuthology.orchestra.run.smithi059.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-19T04:12:46.174 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:12:55.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:12:55.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:12:58.238 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:13:10.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:13:10.241 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:13:13.310 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:13:16.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-19T04:13:16.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T04:13:16.382 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-19T04:13:16.382 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-19T04:13:16.383 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi176 2024-03-19T04:13:16.402 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-19T04:13:16.418 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-19T04:13:16.455 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-03-19T04:13:16.456 INFO:teuthology.nuke:Checking targets against current locks 2024-03-19T04:13:16.484 DEBUG:teuthology.nuke:shortname: smithi059 2024-03-19T04:13:16.485 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T04:13:16.512 DEBUG:teuthology.nuke:shortname: smithi176 2024-03-19T04:13:16.513 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T04:13:16.517 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609439', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 02:20:59.477107', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMzMI8XIydaJGNFQyK09I9223zuTu1APOjRa7nxNh8UVYxaUkDdjFfwzhCSeSg8i83zJoKQWjOtgE59qoumoPSU='} 2024-03-19T04:13:16.538 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609439', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 02:20:59.478516', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOkpD1WsLZAYVh+6AZbSWS/oin4WSKn8n3/VavpxHJfc16KQb/GHcyvbt5NiraEW3cbNG1mGLUAbeI0HcK5TYeM='} 2024-03-19T04:13:16.571 INFO:teuthology.orchestra.console.smithi059:Power off 2024-03-19T04:13:16.572 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-19T04:13:16.598 DEBUG:teuthology.orchestra.console.smithi059:power off output: Chassis Power Control: Down/Off 2024-03-19T04:13:16.598 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T04:13:16.616 INFO:teuthology.orchestra.console.smithi176:Power off 2024-03-19T04:13:16.617 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-19T04:13:16.626 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2024-03-19T04:13:16.639 DEBUG:teuthology.orchestra.console.smithi176:power off output: Chassis Power Control: Down/Off 2024-03-19T04:13:16.639 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T04:13:16.666 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-03-19T04:13:20.629 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T04:13:20.667 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T04:13:20.740 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2024-03-19T04:13:20.778 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-03-19T04:13:24.742 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T04:13:24.779 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T04:13:24.853 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-03-19T04:13:24.890 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-03-19T04:13:24.954 INFO:teuthology.orchestra.console.smithi059:Power off completed 2024-03-19T04:13:28.896 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T04:13:29.007 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is off 2024-03-19T04:13:29.108 INFO:teuthology.orchestra.console.smithi176:Power off completed 2024-03-19T04:13:29.214 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 5745.039198875427 failure_reason: 'Command failed on smithi059 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8ae530-e59d-11ee-95c9-87774f69a715 -e sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 -- bash -c ''ceph orch ps''"' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=89e728a4f41d4131af0d8a9b908bce0c status: fail success: false 2024-03-19T04:13:29.215 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-19T04:13:29.300 INFO:teuthology.run:FAIL